More Android Phone Stuff..

[A_n_g_e_l_a]

[Attachment 74934 - Click to enlarge]
A while ago I posted about decrypting on your Android Phone. That post is still sound but it deals with key finding, using WKS-KEYS, downloading with yt-dlp and decrypting, mainly by hand. We now have scripts and N_m3u8DL-RE to play with

The above linked post is a little out of date now we have N_m3u8DL-RE as a fast all-in-one downloader. But scripts, generally, are more available to do the grunt work of downloading.

[Attachment 74937 - Click to enlarge]
Such as running get_iplayer


[Attachment 74938 - Click to enlarge]
Or other software

With a friendly interface it is now child's play to download to and decrypt videos to your phone. This post is about setting up software on your phone that allows Ubuntu, a variety of Linux, to run scripts as if it were a PC.

Termux is a phone app that allows a Linux emulator to run on your phone. It cannot change anything on your phone system other than files in the storage area that you can see with a file manager - the area known as /sdcard/.

Termux may be installed by getting the apk from github https://github.com/termux/termux-app/releases and selecting arm.v8.
If your phone will not handle arm.v8 or x64 then N_m3u8DL-RE will not work on earlier systems. (A Google Pixel 4XL can be had quite cheaply.)


[Attachment 74933 - Click to enlarge]
So visit Termux on github and download the apk to your phone. Use your phone's file-manager and click on the download to install it. You may need to give system permission to do this.

[Attachment 74939 - Click to enlarge]
I'm going to reproduce here an edited version of the steps to get Termux running on your phone from the VH link at the start

There are several places to get help installing Termux - start looking on reddit.com/r/Termux or https://discord.gg/termux-641256914684084234

To avoid confusion in the description below - phone code looks like this:-

This is written in the terminal emulator screen (Termux)

And your PC instructions typed in a command or terminal window look like this:-

This is typed on your PC

First you'll want to set up house in Termux:

pkg update & upgrade
termux-setup-storage
pkg install openssl openssh

You now have a basic Termux working environment.

Getting stuff on your phone is easy. cable connect and set-up permission for file transfer.

[Attachment 74940 - Click to enlarge]

Android Debug Bridge (adb) is an Android tool supplied by Google. Getting adb set up on your PC is harder on Windows than Linux but use this guide to get adb running IMPORTANT you do need to setup developer permissions on your phone so if yuu need to do that read the XDA article carefully!

adb push ./<some folder>  /sdcard/

sends the whole directory and sub-folders to your phone. We wil use this later.

From here on we need proper access to your phone, from a command window or terminal on your PC, so we can work more easily. We will set up Termux to allow Secure Shell connections (ssh) and openssl was installed to Termux above earlier to facilitate this.

On your phone and in the Termux app set up an access password:-

passwd

find out who you are:

whoami

will respond with something like u0_a100
start the secure shell daemon on your phone with

sshd

Find out your phone IP address :-

ifconfig

Look through the resulting text displayed and find four-triple numbers probably 192.168.1.xxx and make a note of this for use next and in the future.


Now from your PC

ssh -p8022 u0_a100@<phone-IP-address>

You will need your local network address:-

[Attachment 74966 - Click to enlarge]
Find your phone's local network address is in Network Settings.

Now you have a link to your phone it is more easy to configure.
But if you would prefer a wireless connection, after you've setup adb on your PC

adb devices

adb should respond with

* daemon not running; starting now at tcp:5037
* daemon started successfully
List of devices attached
39141XXXG00EC1  device

to use a wireless connection type this to tell adb to use a wireless port

adb tcpip 5555

and connect wirelessly with

adb connect <phone-IP-address>:5555

the cable may now be removed. This feature is recent (few years) and older Android devices may not work.
This may help if you get stuck:- https://www.makeuseof.com/use-adb-over-wifi-android/

NOTE: from now on TYPING IS IN A TERMINAL ON YOUR PC LINKED TO YOUR PHONE
anything you type here will execute on your phone.

It makes sense to have both this guide and your terminal (command) window open as below. In this manner cutting from th guide and pasting in terminal is child's play.


[Attachment 77312 - Click to enlarge]

We will be setting up a Proot using an Ubuntu Linux Distribution. Proot and I quote Google "PRoot allows one to mix transparently the emulated execution of guest programs and the native execution of host programs in the same file-system namespace." that basically means we can run Ubuntu within Termux and Ubuntu may use any Termux programs as its own.

pkg install proot-distro

will install the software to allow PRoot
Once complete we can use proot to install Ubuntu with:-

proot-distro install ubuntu

Wait while it downloads and unpacks. Once finished

proot-distro login ubuntu

will start an ubuntu session.

You can also type directly in Termux too should you wish.

[Attachment 74942 - Click to enlarge]

If you want to end your session type 'exit' to get back to Termux. You may close Termux by typing 'exit' too.

But that's enough for now part 2 will setting up Ubuntu with N_m3u8DL-RE, ffmpeg and all tools needed to download and decrypt will be along soon.

[A_n_g_e_l_a]

Part 2.of More Android Phone Stuff - work in progress

Picking up from:

proot-distro login ubuntu

Start from within Ubuntu to update the list of out-of-date software and upgrade

apt update

apt upgrade

Install some utilities

apt install wget unzip libicu-dev

We are currently in the /root folder of Ubuntu and we can work from here to get most of what we need.

Get N_m3u8D_-RE note ONLY ONE version works

[Attachment 74943 - Click to enlarge]

wget https://github.com/nilaoda/N_m3u8DL-RE/releases/download/v0.1.5.3-beta/N_m3u8DL-RE_Bet...0230323.tar.gz

will download the more commng arm.v8 version Go here for others, and

tar -zxf N_m3u8DL-RE_Beta_linux-arm64_20230628.tar.gz

will unpack to a folder N_m3u8DL-RE_Beta_linux-arm64.

All of the program files will be placed in /usr/local/bin/ in our proot distrobution of Ubuntu.
To do copy

cp N_m3u8DL-RE_Beta_linux-arm64/N_m3u8DL-RE /usr/local/bin/

We will need to give it execute permission on the system, so..

chmod +x /usr/local/bin/N_m3u8DL-RE

will do it. Anything that sits in this folder that needs to run globally will need chmod +x <filename> after placement.

Get shaka-pakager
arm.v8 version is https://github.com/shaka-project/shaka-packager/releases/download/v3.0.1/packager-linux-arm64

wget https://github.com/shaka-project/shaka-packager/releases/download/v3.0.1/packager-linux-arm64

will fetch it. It downloads as a binary directly.
Copy to usr/local/bin and rename

cp packager-linux-arm64 /usr/local/bin/shaka-packager

chmod +x /usr/local/bin/shaka-packager

Get Bento Tools
create a tmp directory and move into it

mkdir tmp; cd tmp

Download Bento Tools

wget https://files.videohelp.com/u/301890/bento4_tools_android.zip

Unpack

unzip bento4_tools_android.zip

Set execute permissions using a wildcard character;

chmod +x *

and copy everything to /usr/local/bin

cp * /usr/local/bin/

Move out of the tmp folder and delete

cd ../; rm -r tmp

Install Mkvtoolnix

apt install mkvtoolnix

Now it would be natural to install ffmpeg using apt here. When I did that I found ffmpeg painfully slow. So I played around with a few alternatives and found that installing ffmpeg from Termux itself is best.
So exit Ubuntu

exit

Now from within Termux type

pkg install ffmpeg

After it has finished log back in to ubuntu

proot-distro login ubuntu

Install get_iplayer (Optional UK only)

apt install software-properties-common
add-apt-repository ppa:m-grant-prg/utils
apt update
apt install get-iplayer

To be continued

[A_n_g_e_l_a]

Setting up Python and installing scripts

[Attachment 74955 - Click to enlarge]
get_iplayer, installed in the last part, running on an Android phone.

We are going to install python and use the env(ironment) feature. a Python Env allows conflicting modules to be installed on a system. Modules are installed into a named env(ironement) and are oblivious to anything outside. It is pretty easy to set up and take care of.

You should be inside your proot-distro , using Ubuntu and in the /root folder.
Check where you are with Print Which Directory pwd

pwd

and it should return /root. If not Change Directory

cd /root

so you are.

Now set up some sub-directories to work in.

mkdir -p ./RE/UK-FTA

will make two folders one inside the other.
cd to RE

cd RE

Install python and a couple of modules using apt

apt install python3 python3-venv python3-pip

Note this system likes to use python3 as a moniker but we can change that later.

When install is complete cd to the UK-FTA folder ready to set up a python env(ironment)

cd UK-FTA

python3 -m venv env

which makes a virtual environment called 'env;.
When complete check it is working.

source env/bin/activate

Your screen prompt should show

(env) root@localhost:~/RE/UK-FTA#

Now we are ready to install pywidevine

(env) pip install pyaml pywidevine

should now install two modules into your env. If not successful go back and check all steps to this point.

cd ~ # /root
mkdir -p keyblob/WVD
cd keyblob

Your CDM client_id.Blob and private.key are going to be copied first to your phone then into the keyblob folder.

From your PC

adb devices

should respond with device and a serial number if you set things up correctly in part 1. If not use you phone connected to your PC and find a way for file manager to copy files across to the phone's Download folder. But note the name changes to client_id.bin and private_key.pem
If using adb move inside the folder with your blob and key on your PC

adb push <blob-name>    /sdcard/Download/client_id.bin

adb push <key-name>    /sdcard/Download/private_key.pem

From your phone and inside the keyblob folder

mv /sdcard/download/client_id.bin .

Don't miss the full-stop in that command! Do the same for private key

mv /sdcard/download/private_key.pem .

Create a WVD file

[Attachment 74956 - Click to enlarge]
Change the names your your blob and key in this command

pywidevine create-device -c client_id.bin -k private_key.pem -t ANDROID -l 3 -o WVD

Should complete without fuss. Now cd to the WVD folder and use 'ls' to list the file contents,
You should find a long name refecting the orgin of the CDM. We will simplify this to mbox.wvd.
Move the file to a new name start by typing mv and the first characters of the name and use auto-complete. My key for that is 'tab'.

mv <wvd-filename> mbox.wvd

note the location /root/keyblob/WVD/mbox.wvd, if ever scripts ask where your wvd file it is /root/keyblob/WVD/mbox.wvd

Installing optional scripts
You now have a working python environment and any scripts that work on a PC should work here.
The only change needed to to note that N_m3u8DL-RE is an earlier version and --no-logs in the N_m3u8DL-RE command will crash it. Edit out --no-log

In any folder where you suspect N_m3u8DL-RE is being use of this command will find files that need editing

grep -r --no-log .

will show if any files have the command present and will need editing.
Nano is the editor

nano <file_path/filename>

to open it. Ctrl +W --no-log to find it. Then put # in front to edit it out.
Any scripts should work now that use N_m3U8DL-RE. yt-dlp can be installed too. (My UK- FTA installs yt-dlp using pip )

If you wish to use my UK-FTA scripts I have an edited version made more suitable for a phone screen.

cd /root/RE/UK-FTA

wget https://files.videohelp.com/u/301890/phone-uk-fta_13.4.zip

unzip UK-FTA.zip

should inflate the zip to the current directory
Install the required module

[Attachment 74957 - Click to enlarge]
Still within the python (env) - (if not restart it)

pip install -r requirements.txt

Start the script

python3 menu.py

If all has gone well you should see

[Attachment 74958 - Click to enlarge]

Before going any further let us close the env and make a few simple changes to make working and typing on the phone easier.

deactivate

will close env

cd /root

nano .bashrc

(Note the full stop)
We are going to add a few aliases such that we can type just one letter to change to a UK-FTA and call python3 just by typing 'py'.

[Attachment 74959 - Click to enlarge]

paste this to the very end of the .bashrc file

alias u='cd /root/RE/UK-FTA/'
alias py='python3'
alias env='cd /root/RE/UK-FTA; source env/bin/activate'
alias r='cd /root/RE'
alias d='deactivate; echo "Closing Python env"'

Ctrl+s saves and ctrl+x exits nano editor.
Now logout from Ubuntu and login again to activate the aliases we have set

exit

drops you back to a $ prompt in Termux

proot-distro login ubuntu

re-enters Ubuntu now type env and the prompt changes to show env and you've also changed to UK-FTA folder.
This time 'py' can be use instead of python3 and use auto-complete to avoid typing too much.

On your phone screen look for the two opposing arrows in the region just above the keyboard - these arrows are the auto-complete key.
The other extra Termux functions above the phone keyboard have up-down/left-right controls for moving a cursor. CTRL is sticky and waits for the next key C or D, or whatever, to allow emulation of a PC keyboard. Everything you might need is there for keyboard entry. Scripts ideally would try to minimise keyboard entry.

[Attachment 74960 - Click to enlarge]

[Attachment 74961 - Click to enlarge]

[Attachment 74962 - Click to enlarge]

[Attachment 74964 - Click to enlarge]

[Attachment 74963 - Click to enlarge]

[Attachment 74965 - Click to enlarge]
UK-FTA should run 'out-of-the-box as all configs follow the install pattern of placing and naming files. If you made changes then maybe the config file will need editing in nano

Any videos downloaded by UK-FTA will appear in the sdcard Download folder so it is accessible to apps like VLC or Casting apps.

Let me be clear any python script should be able to run on Ubuntu. If you use scripts that have need of module imports first create a new folder from RE move into that new folder and create a NEW venv env1 and activate with source env1/bin/activate and install any modules those scripts need. Deactivate when finished.. In this way may author's scripts may happily exist on a system each not affecting the other.

Finally. Casting to a TV is an obvious use. Or using your phone to watch videos during a commute to work. But is is also a backup to download video if your main machine may be out of action.

That's all I know.

[A_n_g_e_l_a]

Freevine
Freevine seems to be a popular downloader and it was very quick to get it working on my phone.

[Attachment 74984 - Click to enlarge]

[Attachment 74983 - Click to enlarge]
Close env if open: We can use the shortcut 'd' or 'deactivate'

d
cd /root
mkdir FV
cd FV
wget https://github.com/stabbedbybrick/freevine/archive/refs/heads/main.zip
unzip main.zip
mv main freevine-main

renaming main to freevine-main makes it less anonymous.
Next set up an env for the new scripts first move into the scripts top level folder and then create a new env - I called mine 'frenv'

cd freevine-main
py -m venv frenv
source frenv/bin/activate

You may decide to set up another alias to start frenv from a single typed character.

There is one file that will kill N_m3u8DL-RE and needs editing out:-

nano ./utils/args.py

from within nano, CTRL +w opens search. Enter --no-log RETURN and comment out the highlighted line

[Attachment 74985 - Click to enlarge]

Copy your CDM to Freevine
from within freevine-main

cd utils
mkdir wvd
cd wvd
cp -r /root/RE/keyblob/* .

(Note the full stop!)

Now Freevine will run if the modules it needs are installed.

pip install -r requirements.txt

And wait for it to finish.

Freevine is a command line tool and instead of typing urls they can be, happily, pasted. Termux allows pasting anywhere below the current line and automatically places it for you. So copy a test url - I chose BBC iplayer

py freevine --episode S08E01 https://www.bbc.co.uk/iplayer/episodes/p01s711r/shetland?seriesId=m001s15m

Which downloaded and saved to a folder beneath freevine-main. I guess there will be a configuration file to tweak to save to /sdcard/ so your videos are more accessible.
So Freevine works on a mobile phone. There is a need to type and copy and paste but no insurmountable difficulties.
There is one gotcha. N_m3u8DL-RE sometimes leaves the screen so that type does not appear. Just type 'reset' and ENTER to bring back normal behaviour. This effect happens on Linux

As a side note: I noticed some windows exe files being distributed with the Freevine package.

[Attachment 74986 - Click to enlarge]
Windows users be careful.

Get_iplayer
I thought I would have issues with get_iplayer but it does what it does on a mobile phone very well.


[Attachment 74987 - Click to enlarge]
As well as downloading video get_iplayer will fetch radio broadcasts too.

[Attachment 74988 - Click to enlarge]

Finally!! My install uses 2.21 Gb of space on my phone. You'll be wise to set Termux's 'App Battery Usage' to unrestricted to allow it to work unhindered.
In use I had a clash with the keyword env set as an alias in .bashrc . I solved it by renaming the alias to envo. Happy Days.

[peterb999]

Looks intesting, thanks Angela. Is your thinking that you could be out and about and download and watch the shows? I'll give it a go.

[A_n_g_e_l_a]

Looks intesting, thanks Angela. Is your thinking that you could be out and about and download and watch the shows? I'll give it a go.

Many use-cases really. With my phone nearby it is sometimes more convenient than firing up the laptop. I imagine some might like fast wifi at work to do their download. Casting something you want to watch without adverts but not keep - so a quick download and immediate cast to the TV would work well. Use while travelling, use while someone else uses the home computer. Kids favourite cartoons for that family posh-restaurant meal to keep them a little quieter. Loads of uses if you want. None obligtory

[sipho]

This is very good A_n_g_e_l_a.

I need to digest all of this as I'm not a big phone user other than talking on it and WhatsApp. I can see the benefits though.

[A_n_g_e_l_a]

Freevine works on a mobile phone https://forum.videohelp.com/threads/412382-More-Android-Phone-Stuff#post2713017

[A_n_g_e_l_a]

get_iplayer leveraged to play nicely with N_m3u8DL-RE on a mobile

:
[Attachment 75149 - Click to enlarge]:
[Attachment 75150 - Click to enlarge]:
[Attachment 75146 - Click to enlarge]:
[Attachment 75147 - Click to enlarge]Get_iplayer fetches the file url for download by N_m~:
[Attachment 75148 - Click to enlarge]

[peterb999]

Looks really good. I didn't realise you'd added BBC to the FTA downloader.

[A_n_g_e_l_a]

Looks really good. I didn't realise you'd added BBC to the FTA downloader.

I haven't.

[peterb999]

Ah, I thought I'd missed an update or something.

[A_n_g_e_l_a]

Ah, I thought I'd missed an update or something.

I'm not sure FTA needs a BBC downloader. There are tools already out there and there is no encryption. So anyone can download from the BBC without a problem.
Yt-dlp downloads 720p from a URL; get_iplayer will download at 1080p.

On a phone screen, typing is a issue, so it makes sense, to me, to use a script.

I posted this as I'd never seen anyone using get_iplayer to find download URLs for another downloader.

[peterb999]

I really like the search bit of your FTA script, it's useful not even having to go to the website.

I'm going to add it to mine just to see if I can work out how you do it. It is great fun doing stuff like this and it is good to not be spoon-fed all the time and I really appreciate all you do for the forum.

[sipho]

I'm pretty sure this is what peterb999 was referring to.


[Attachment 75176 - Click to enlarge]

[A_n_g_e_l_a]

Update to Phone-UK-FTA https://files.videohelp.com/u/301890/Phone-UK-FTA_2.zip includes BBC service and channel 5 HMAC updater is refreshed.
To use the BBC links you will need to install get_iplayer (and prove working from the command line). Installation of get_iplayer is detailed in Part two, above, as a package install in Termux

[A_n_g_e_l_a]

Updated to correct bug. Script was looking for program_title, bbc_channel, program_ID returned from BBC.
Sometimes BBC send program_title, extra_unexpected_data, bbc_channel, program_ID. The extra unexpected data caused the script to fail. It is now conflated with program_title

https://files.videohelp.com/u/301890/Phone-UK-FTA_3.zip

[A_n_g_e_l_a]

BBC search api api-key has changed. If you use the BBC links from menu.py then
edit line 162 of ukft/bbc_dl/getBBCLinks.py and replace everything from and including 'apikey= to
the line end with the following:-

apikey=D2FgtcTxGqqIgLsfBWTJdrQh2tVdeaAp&seqId=0582e0f0-b911-11ee-806c-11c6c885ab56

so the whole line looks like this:

url = f"https://search.api.bbci.co.uk/formula/iplayer-ibl-root?q={search}&apikey=D2FgtcTxGqqIgLsfBWTJdrQh2tV deaAp&seqId=0582e0f0-b911-11ee-806c-11c6c885ab56"

[A_n_g_e_l_a]

I've added the above BBC update and a few other bits to this new release. Remember to save your original configs/config.py to somewhere while inflating the zip.


[Attachment 76364 - Click to enlarge]

https://files.videohelp.com/u/301890/phone-UKFTA-4.zip

[sipho]

Thank you!

[A_n_g_e_l_a]

The last service addition has been through a bit of turmoil and I've written a downloader that will resume if the download fails. There is a new python module 'tqdm' to install.
Download progress is less pretty than the last one but I couldn't get Rich.progress to work.

I fail to remember to keep My5 HMAC keys updated. Maybe I need to do a HMAC lookup on start - maybe coming next.

https://files.videohelp.com/u/301890/phone-UKFTA-5.zip

[A_n_g_e_l_a]

Updates to the last service addition: to add new link. Old link can be set back by direct script edit.
Also My5 now updates keys automatically on each use.

https://files.videohelp.com/u/301890/phone-UKFTA-6.zip

[A_n_g_e_l_a]

Updated Vadapav only. Now has extended timeouts for all https traffic. Reworked the data displayed to only show filenames. Plus a few bugs squashed.

So in the meantime just releasing the vadapav.py phone version. Copy it into the UK-FTA/ukfta/vadapav/ folder on your phone and replace the existing vadapav.py with this:-
https://files.videohelp.com/u/301890/phone_vada.py
Be sure to rename it vadapav.py

[A_n_g_e_l_a]

Tweaks, bug fixes and update to vadapav mainly

[Attachment 76711 - Click to enlarge]

https://files.videohelp.com/u/301890/phone-UKFTA-7.zip

[A_n_g_e_l_a]

A new update to sync to the PC Version. Version number jumps as a consequence.

Changes have been made to vadapav so it uses the configuration file to find its storage requirements. Vadapav.mov is a file reposository with film and TV media. Some media is 4k and a 2160 search will find it. It probably shouldn't be in an FTA list but I use use it and it is convenient to access here.

Channel 4 updated with my spoof data.

Channel 5 now updates its own HMAC secret and AES keys. It will update the first time my5 is called and once every three days, but only when my5 is used. It shouldn't need intervention. (I only use my5 rarely but I got fed up running my5 and having it fail with out of date keys.)

The menu has been adjusted to reflect changes getting my5 HMAC secret and AES key.

NOTE: It is important to replace the total existing UK-FTA folder as there have been structural changes. Run config from the menu to set up storage and the wvd location. (Maybe copy those from the current config before installation of the new version?)

Requirements.txt has been updated and reduced. The need for get_iplayer, as a helper program, has been added to the top level README.md

The last phone version had 25 downloads; either that is because it had a Channel4 script that worked or because more of you are finding out just how useful it is having it on your phone. With Channel4 fixed on the PC version I'll soon know!


https://files.videohelp.com/u/301890/phone-uk-fta_13.zip

[PSXman_uk]

[Attachment 77460 - Click to enlarge]

Latest version of termux doesnt like the commands ive searched the folders to do the commands


pkg update & upgrade
termux-setup-storage
pkg install openssl openssh

Ignore sorry had to change repository to add these

[A_n_g_e_l_a]

Small bug fix to vadapav mainly. If you haven't noticed any bugs then wait till the next release.
https://files.videohelp.com/u/301890/phone-uk-fta_13.4.zip

[PSXman_uk]

Re read and fixed issue was sdcard when you adb to it is the download folder on the device not sdcard itself

[A_n_g_e_l_a]

Re read and fixed issue was sdcard when you adb to it is the download folder on the device not sdcard itself

That is not standard behaviour, adb push <filename> /sdcard/ is expected to send filename to the base of your personal internal memory where Download is a folder.

If you want to place something in Download

adb push <filename> /sdcard/Download/

Unless you have a phone with a slot for an extra card memory, and I cannot remember how those are accessed from adb

[PSXman_uk]

Thank you for the tutorial A_n_g_e_l_a i have now got it all working on the phone only thing i had to do different was to copy the args.py to sdcard then edit and put it back going to do the same with config.yaml to change the download dir when I can figure out exactly what the sdcard directory is as sdcard/download is the download dir on the device not the sdcard


Drop out of lnux

cd ~/storage

ls -la
/storage/6E77-12EB/ is my sdcard name for example

Thanks again for the Tutorial

Still cant see it within the linux enviroment though