diaojiaolou
/
Deep-Live-Cam
mirror of https://github.com/hacksider/Deep-Live-Cam.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Deep-Live-Cam/google-colab/DeepLive_Google_Colab.ipynb

1503 lines
49 KiB
Plaintext
Raw Normal View History

google colab
2024-08-16 11:47:09 +08:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "U3Fc3OXuDeqP"
},
"source": [
"# **Install InsightFace and Dependencies:**\n",
"\n",
" Run the following code to install the required packages.\n",
"\n",
" Note: This installation varies and depends on your cuda and cudnn version. Incase of face swapping issue refer to the link in the debug session to know the version to install."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"id": "D-UgHjSFBE9d"
},
"outputs": [],
"source": [
"#!pip install onnx==1.16.0\n",
"!pip install onnxruntime-gpu==1.18.0 --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/\n",
"!pip install insightface #==0.7.3\n",
"#!pip install onnxruntime==1.18.0\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Q81Sf-CFarjq"
},
"outputs": [],
"source": [
"!apt-get install -y ffmpeg"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "GYFbuuCdvTaO"
},
"source": [
"### DeepFakeLive\n",
"Follow the steps below to create the necessery directories for the project"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"id": "_MDoAtpyvV_l"
},
"outputs": [],
"source": [
"!cd /content\n",
"!rm -rf *"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"id": "RzPl9sPVgSoP"
},
"outputs": [],
"source": [
"!mkdir deepfakecollab"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "lJJbUDokN0bF",
"outputId": "a2ef8b6e-97a8-4ef2-f986-e34e6eb1252e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/content/deepfakecollab\n"
]
}
],
"source": [
"cd deepfakecollab"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "ZUzAF6L2N6__"
},
"outputs": [],
"source": [
"!mkdir Scripts"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ziSLHxab_j0h"
},
"outputs": [],
"source": [
"!ffmpeg -version"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0rTCd2R1uw7A"
},
"source": [
"### [Optional] FRP\n",
"\n",
"Follow the steps below to setup FRP. You will also need to host the FRPS on your VPS (free).\n",
"\n",
"Note: You cannot run both FRP and ngrok together. Its ether FRP and ngrok"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ckp_dM52N9K3"
},
"outputs": [],
"source": [
"!touch Scripts/get_frs.sh\n",
"\n",
"getfrs = \"\"\"#!/usr/bin/env bash\n",
"\n",
"# Check if frpc is installed\n",
"command -v frpc >/dev/null 2>&1\n",
"if [[ $? -ne 0 ]]; then\n",
" echo \"frpc is not found, installing...\"\n",
" wget -q -nc https://github.com/fatedier/frp/releases/download/v0.59.0/frp_0.59.0_linux_amd64.tar.gz\n",
" tar -xzf frp_0.59.0_linux_amd64.tar.gz\n",
" echo \"Done!\"\n",
"fi\"\"\"\n",
"with open('Scripts/get_frs.sh', 'w') as f:\n",
" f.write(getfrs)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "w21kxVRmODFZ"
},
"outputs": [],
"source": [
"!touch Scripts/open_tunnel_frs.sh\n",
"\n",
"getfrs = \"\"\"#!/usr/bin/env bash\n",
"\n",
"cmd=\"frp_0.59.0_linux_amd64/frpc -c frp_0.59.0_linux_amd64/frpc.toml\"\n",
"\n",
"kill -9 $(ps aux | grep $cmd | awk '{print $2}') 2> /dev/null\n",
"\n",
"echo Opening tunnel\n",
"$cmd\"\"\"\n",
"with open('Scripts/open_tunnel_frs.sh', 'w') as f:\n",
" f.write(getfrs)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "tmOY654Mia79"
},
"outputs": [],
"source": [
"!chmod +x Scripts/get_frs.sh\n",
"!chmod +x Scripts/open_tunnel_frs.sh"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Br-P5pkOwFg1",
"outputId": "663dae11-27c6-4329-a768-58faa1dd14e9"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"frpc is not found, installing...\n",
"Done!\n"
]
}
],
"source": [
"!Scripts/get_frs.sh"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9n-qESinvM22"
},
"source": [
"### [Optional] Ngrok\n",
"\n",
"Follow to Setup Ngrok.\n",
"\n",
"Note: You need an API key from Ngrok to use tcp for free you would need to add a billing details to their platform"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "S9U6x85DvTTV"
},
"outputs": [],
"source": [
"!touch Scripts/get_ngrok.sh\n",
"\n",
"getfrs = \"\"\"#!/usr/bin/env bash\n",
"\n",
"# Check if frpc is installed\n",
"command -v frpc >/dev/null 2>&1\n",
"if [[ $? -ne 0 ]]; then\n",
" echo \"ngrok is not found, installing...\"\n",
" wget -q -nc https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz\n",
" tar -xzf ngrok-v3-stable-linux-amd64.tgz\n",
" echo \"Done!\"\n",
"fi\"\"\"\n",
"with open('Scripts/get_ngrok.sh', 'w') as f:\n",
" f.write(getfrs)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"id": "ZYKKKRZSwA82"
},
"outputs": [],
"source": [
"!touch Scripts/open_tunnel_ngrok.sh\n",
"\n",
"getfrs = \"\"\"#!/usr/bin/env bash\n",
"\n",
"cmd=\"./ngrok start --all --config ngrok.conf\"\n",
"\n",
"kill -9 $(ps aux | grep $cmd | awk '{print $2}') 2> /dev/null\n",
"\n",
"echo Opening tunnel\n",
"$cmd\"\"\"\n",
"with open('Scripts/open_tunnel_ngrok.sh', 'w') as f:\n",
" f.write(getfrs)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "A2FQrmDQwW__"
},
"outputs": [],
"source": [
"!chmod +x Scripts/get_ngrok.sh\n",
"!chmod +x Scripts/open_tunnel_ngrok.sh"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "WoNs1gSiweBm",
"outputId": "9e24a0a6-b50b-49f6-a928-2c004917927e"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ngrok is not found, installing...\n",
"Done!\n"
]
}
],
"source": [
"!Scripts/get_ngrok.sh"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"id": "uCPALISZwo3K"
},
"outputs": [],
"source": [
"# Paste your authtoken here in quotes\n",
"authtoken = \"\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uVpDmuTmwtO9"
},
"source": [
"Set your region\n",
"\n",
"Code | Region\n",
"--- | ---\n",
"us | United States\n",
"eu | Europe\n",
"ap | Asia/Pacific\n",
"au | Australia\n",
"sa | South America\n",
"jp | Japan\n",
"in | India"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"id": "MM-UUANCwwqo"
},
"outputs": [],
"source": [
"# Set your region here in quotes\n",
"region = \"eu\""
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JmNGrasvwnVX"
},
"source": [
"### Create Model Folder\n",
"\n",
"Create and download into the model folder"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"id": "rYAgrA75wv4U"
},
"outputs": [],
"source": [
"!mkdir -p Model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "EU1ZXGlGw8Rb"
},
"outputs": [],
"source": [
"!wget https://github.com/facefusion/facefusion-assets/releases/download/models/inswapper_128_fp16.onnx -P /content/deepfakecollab/Model\n",
"!wget https://huggingface.co/hacksider/deep-live-cam/resolve/main/GFPGANv1.4.pth -P /content/deepfakecollab/Model"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sNHGT2FjjZiK"
},
"source": [
"### [Ignore] DeBuggin (Run only when its necessery)\n",
"\n",
"Debugging to ensure cuda was used. if not check the version of the cudnn, edit the install of the onnxruntime package to install the right version. https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ZQr-gWl6Ibw7",
"outputId": "ef468d22-a4c5-4f73-932b-dea535cc25ed"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"#define CUDNN_MAJOR 8\n",
"#define CUDNN_MINOR 9\n",
"#define CUDNN_PATCHLEVEL 6\n",
"--\n",
"#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)\n",
"\n",
"/* cannot use constexpr here since this is a C-only file */\n"
]
}
],
"source": [
"!cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR -A 2"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "RZ9r-s_9Kdha",
"outputId": "b8778f0e-f10a-4725-cac7-67a435992100"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.18.0\n"
]
}
],
"source": [
"import onnxruntime as rt\n",
"print(rt.__version__)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Stwb4OHv_hs0",
"outputId": "41d0963a-31df-4fea-f80b-cbebef56d07c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Active providers: ['CUDAExecutionProvider', 'CPUExecutionProvider']\n"
]
}
],
"source": [
"# Ensure the GPU providers are set explicitly\n",
"ort_session = rt.InferenceSession(\n",
" \"/content/deepfakecollab/Model/inswapper_128_fp16.onnx\",\n",
" providers=[\"CUDAExecutionProvider\", \"CPUExecutionProvider\"]\n",
")\n",
"\n",
"# Verify the active provider again\n",
"print(\"Active providers:\", ort_session.get_providers())\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "DTbmyO-CK-8b"
},
"source": [
"### Create the Colab Server\n",
"\n",
"In your Google Colab notebook, set up a TCP server that will receive frames, process them using the FACE_SWAPPER model, and send back the results."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"id": "V_Luv36jjcmH"
},
"outputs": [],
"source": [
"import socket\n",
"import cv2\n",
"import numpy as np\n",
"import insightface\n",
"import threading\n",
"import torch\n",
"import onnxruntime\n",
"from typing import Any\n",
"from insightface.app.common import Face\n",
"import matplotlib.pyplot as plt\n"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "P4xE5bpeosP_",
"outputId": "7562a24e-ca5f-491e-884e-ce43b2ca0325"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['CUDAExecutionProvider']\n"
]
}
],
"source": [
"# Check if CUDA is available\n",
"if 'CUDAExecutionProvider' in onnxruntime.get_available_providers():\n",
" providers = ['CUDAExecutionProvider']\n",
"elif 'TensorrtExecutionProvider' in onnxruntime.get_available_providers():\n",
" providers = ['TensorrtExecutionProvider']\n",
"else:\n",
" providers = ['CPUExecutionProvider']\n",
"print(providers)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0OFrqUjgz7zb"
},
"outputs": [],
"source": [
"\n",
"FACE_SWAPPER = None\n",
"FACE_ANALYSER = insightface.app.FaceAnalysis(name='buffalo_l', providers=providers)\n",
"FACE_ANALYSER.prepare(ctx_id=0, det_size=(640, 640))\n",
"THREAD_LOCK = threading.Lock()\n",
"Frame = np.ndarray[Any, Any]"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"id": "ZbPSlDq_zJLj"
},
"outputs": [],
"source": [
"def get_face_swapper() -> Any:\n",
" global FACE_SWAPPER\n",
"\n",
" with THREAD_LOCK:\n",
" if FACE_SWAPPER is None:\n",
" model_path = \"/content/deepfakecollab/Model/inswapper_128_fp16.onnx\"\n",
" FACE_SWAPPER = insightface.model_zoo.get_model(model_path, providers=['CUDAExecutionProvider'])\n",
" return FACE_SWAPPER"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"id": "EnNJl5kDOjrL"
},
"outputs": [],
"source": [
"def swap_face(source_face: Face, target_face: Face, temp_frame: Frame) -> Frame:\n",
" return get_face_swapper().get(temp_frame, target_face, source_face, paste_back=True)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"id": "y0VduuqyaA8R"
},
"outputs": [],
"source": [
"def get_face_analyser() -> Any:\n",
" #FACE_ANALYSER.prepare(ctx_id=0, det_size=(640, 640))\n",
" return FACE_ANALYSER"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"id": "ZTHjQ0p2aI9c"
},
"outputs": [],
"source": [
"def get_one_face(frame: Frame) -> Any:\n",
" face = get_face_analyser().get(frame)\n",
" try:\n",
" return min(face, key=lambda x: x.bbox[0])\n",
" except ValueError:\n",
" return None\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"id": "ygR97Za2aTVO"
},
"outputs": [],
"source": [
"def get_many_faces(frame: Frame) -> Any:\n",
" try:\n",
" return get_face_analyser().get(frame)\n",
" except IndexError:\n",
" return None"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"id": "05SG37Fbwip1"
},
"outputs": [],
"source": [
"def process_frame(source_face: Face, temp_frame: Frame,manyface: bool) -> Frame:\n",
"\n",
" if manyface:\n",
" many_faces = get_many_faces(temp_frame)\n",
" if many_faces:\n",
" for target_face in many_faces:\n",
" temp_frame = swap_face(source_face, target_face, temp_frame)\n",
" else:\n",
" target_face = get_one_face(temp_frame)\n",
" if target_face:\n",
" temp_frame = swap_face(source_face, target_face, temp_frame)\n",
" return temp_frame"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"id": "znqUf8Gnq4dJ"
},
"outputs": [],
"source": [
"# Input and output ports for communication\n",
"local_in_source = 5555\n",
"local_in_temp = 5556\n",
"local_out_frame = 5557"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "FehUyVNvGGgZ"
},
"source": [
"### [Optional] For Live Streaming\n",
"For Live Streaming from webcam run this cell but for just image swap run the next cell. \n",
"\n",
"Note: Don't run both cells at same time. You should either run this or the one below\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "VYQRVPsdxizn",
"outputId": "def84ce6-46cf-401f-8e60-f1f0216b2975"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"PULL one socket bound to tcp://127.0.0.1:5555\n",
"OutputStream ffmpeg bound to tcp://127.0.0.1:5557?listen\n",
"InputStream ffmpeg bound from tcp://127.0.0.1:5556?listen\n"
]
}
],
"source": [
"import zmq\n",
"import threading\n",
"import cv2\n",
"import numpy as np\n",
"import msgpack\n",
"import queue\n",
"import time\n",
"import zlib\n",
"from tqdm import tqdm\n",
"import subprocess\n",
"from collections import deque\n",
"#import matplotlib.pyplot as plt\n",
"\n",
"def create_demo_image():\n",
" # Create a demo image (e.g., a solid color or pattern)\n",
" demo_image = np.zeros((540, 960, 3), dtype=np.uint8) # Black image\n",
" cv2.putText(demo_image, 'Demo Image', (50, 240), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)\n",
" return demo_image\n",
"\n",
"def pull_socket(local_in_port):\n",
" context = zmq.Context()\n",
" socket = context.socket(zmq.REP)\n",
" socket.setsockopt(zmq.RCVHWM, 100000)\n",
" socket.setsockopt(zmq.LINGER, 0)\n",
" address = f\"tcp://127.0.0.1:{local_in_port}\"\n",
" socket.bind(address) # Binding to a different local port\n",
" print(f\"PULL one socket bound to {address}\")\n",
" return socket\n",
"\n",
"\n",
"# Compress image\n",
"def compress_image(image, quality=95):\n",
" # Set the JPEG quality parameter\n",
" encode_param = [int(cv2.IMWRITE_JPEG_QUALITY), quality]\n",
" # Encode the image as a JPEG\n",
" result, encimg = cv2.imencode('.jpg', image, encode_param)\n",
"\n",
" if not result:\n",
" raise Exception(\"Image encoding failed\")\n",
"\n",
" # Decode the encoded image back to an image format\n",
" decimg = cv2.imdecode(encimg, 1)\n",
" return decimg\n",
"# Decompress image\n",
"def decompress_image(encimg):\n",
" image = cv2.imdecode(np.frombuffer(encimg, np.uint8), cv2.IMREAD_COLOR)\n",
" return image\n",
"\n",
"# Global variables\n",
"frames_array=deque(maxlen=2000)\n",
"source_frame = None\n",
"is_manyFace = None\n",
"frameSize = '960x540'\n",
"fps = None\n",
"\n",
"#functions\n",
"def pull_worker(pull_socket):\n",
" global source_frame,is_manyFace,frameSize,fps\n",
" while True:\n",
" try:\n",
" # Receive the JSON with total chunks\n",
" meta_data_json = pull_socket.recv_json()\n",
" #print(meta_data_json)\n",
" total_chunk = meta_data_json['total_chunk']\n",
" # Send acknowledgment for metadata\n",
" pull_socket.send_string(\"ACK\")\n",
" # Receive the array bytes\n",
" source_array_bytes =b''\n",
" for i in range(total_chunk):\n",
" chunk = pull_socket.recv()\n",
" source_array_bytes += chunk\n",
" pull_socket.send_string(f\"ACK {i + 1}/{total_chunk}\")\n",
"\n",
"\n",
" end_message = pull_socket.recv()\n",
" if end_message == b\"END\":\n",
" pull_socket.send_string(\"Final ACK\")\n",
"\n",
" # Deserialize the bytes back to an ndarray\n",
" source_array = np.frombuffer(source_array_bytes, dtype=np.dtype(meta_data_json['dtype_source'])).reshape(meta_data_json['shape_source'])\n",
"\n",
" #plt.imshow(source_array[:, :, ::-1])\n",
" #plt.show()\n",
" #frame_queue.append([\"source\", source_array])\n",
" source_frame = source_array\n",
" is_manyFace = meta_data_json['manyface']\n",
" frameSize = meta_data_json['size']\n",
" fps = meta_data_json[\"fps\"]\n",
"\n",
"\n",
" #process_queue.put((\"source\", source_array))\n",
" break\n",
" except zmq.Again:\n",
" # Sleep briefly to avoid busy-waiting\n",
" time.sleep(0.01)\n",
" except Exception as e:\n",
" print(f\"Error: {e}\")\n",
"def pull_worker_two(local_in_temp):\n",
" ffmpeg_receive_command = [\n",
" 'ffmpeg',\n",
" '-i',f'tcp://127.0.0.1:{local_in_temp}?listen',\n",
" '-f','rawvideo',\n",
" '-pix_fmt','bgr24',\n",
" '-s','960x540',\n",
" 'pipe:1'\n",
" ]\n",
" ffmpeg_receive_process = subprocess.Popen((ffmpeg_receive_command), stdout=subprocess.PIPE)\n",
" timefame =1/25\n",
" print(f\"InputStream ffmpeg bound from tcp://127.0.0.1:{local_in_temp}?listen\")\n",
" global source_frame,is_manyFace\n",
" while True:\n",
" try:\n",
"\n",
" # Receive the JSON with total chunks\n",
" # Read decoded frame from FFmpeg\n",
" raw_frame = ffmpeg_receive_process.stdout.read(960 * 540 * 3)\n",
" if not raw_frame:\n",
" break\n",
" framex = np.frombuffer(raw_frame, dtype=np.uint8).reshape((540, 960, 3))\n",
" #print(framex)\n",
" source_array = source_frame\n",
" is_many_face = is_manyFace\n",
"\n",
" if source_array is not None:\n",
" processed_array = process_frame(get_one_face(source_array),framex,is_many_face)\n",
" #plt.imshow(processed_array[:, :, ::-1])\n",
" #plt.show()\n",
" frames_array.append(processed_array)\n",
"\n",
" time.sleep(timefame)\n",
"\n",
"\n",
" except zmq.Again:\n",
" # Sleep briefly to avoid busy-waiting\n",
" time.sleep(0.01)\n",
" except Exception as e:\n",
" print(f\"Error: {e}\")\n",
"def push_worker(local_out_frame):\n",
" source_array = None\n",
" temp_array = None\n",
" is_many_face = None\n",
" global source_frame,is_manyFace,frameSize,fps\n",
" print(f\"OutputStream ffmpeg bound to tcp://127.0.0.1:{local_out_frame}?listen\")\n",
" ffmpeg_encode_command = [\n",
" 'ffmpeg',\n",
" '-f', 'rawvideo',\n",
" '-pix_fmt', 'bgr24',\n",
" '-s', '960x540',\n",
" '-r', '5',\n",
" '-i', 'pipe:',\n",
" #'-vf', 'fps=5',\n",
" #'-c:v', 'libx264',\n",
" #'-probesize', '32',\n",
" #'-analyzeduration', '0',\n",
" '-f', 'mpegts',\n",
" f'tcp://127.0.0.1:{local_out_frame}?listen'\n",
" ]\n",
" timefame =1/5\n",
"\n",
" ffmpeg_encode_process = subprocess.Popen((ffmpeg_encode_command), stdin=subprocess.PIPE)\n",
"\n",
"\n",
" demo_image = create_demo_image()\n",
" frames_to_skip = 200 # Number of frames to skip to reduce delay\n",
" frame_count =0\n",
" wait_frame = 100\n",
" try:\n",
" while True:\n",
"\n",
" # Get the processed array and metadata from the queue\n",
" if len(frames_array)>wait_frame:\n",
"\n",
" while len(frames_array)>7:#frame_count<frames_to_skip:\n",
" frames_array.popleft()\n",
" frame_count+=1\n",
" if len(frames_array)==7:\n",
" wait_frame = 5\n",
"\n",
" temp_array = frames_array.popleft()#process_queue.get()\n",
" source_array = source_frame\n",
" is_many_face = is_manyFace\n",
" #print(source_array,temp_array)\n",
" if source_array is not None and temp_array is not None:\n",
" #print(\"frames_array,\",len(frames_array))\n",
" #processed_array =process_frame(get_one_face(source_array),temp_array,is_many_face)\n",
" processed_array_bytes = temp_array.tobytes()#processed_array.tobytes()#temp_array.tobytes()\n",
"\n",
" ffmpeg_encode_process.stdin.write(processed_array_bytes)\n",
" #push_socket.send(zlib.compress(processed_array_bytes))\n",
" source_array = None\n",
" temp_array = None\n",
" is_many_face = None\n",
" else:\n",
" framex = demo_image\n",
" # Write the frame to FFmpeg for encoding and streaming\n",
" ffmpeg_encode_process.stdin.write(framex.tobytes())\n",
" time.sleep(timefame)\n",
" finally:\n",
" #ffmpeg_receive_process.terminate()\n",
" ffmpeg_encode_process.terminate()\n",
"\n",
"# Create sockets\n",
"pull_socket = pull_socket(local_in_source)\n",
"\n",
"# Run both workers in separate threads\n",
"# Start the pull worker thread\n",
"pull_thread = threading.Thread(target=pull_worker, args=(pull_socket,))\n",
"pull_thread.start()\n",
"# Start the push worker thread\n",
"pull_thread_two = threading.Thread(target=pull_worker_two, args=(local_in_temp,))\n",
"pull_thread_two.start()\n",
"# Start the push worker thread\n",
"push_thread = threading.Thread(target=push_worker,args=(local_out_frame,))\n",
"push_thread.start()\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Eb_cbyoaHKrT"
},
"source": [
"### [Optional] For Image Swap\n",
"For image swapping run this cell.\n",
"\n",
"Note: You can not run both cell at same time. Either This or The cell Above (For Live Streaming)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "A_nrCfTpHWS4",
"outputId": "10038760-2787-44ed-83fb-35e74a757205"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"PULL one socket bound to tcp://127.0.0.1:5555\n",
"PULL one socket bound to tcp://127.0.0.1:5556\n",
"PUSH socket bound to tcp://127.0.0.1:5557\n"
]
}
],
"source": [
"import zmq\n",
"import threading\n",
"import cv2\n",
"import numpy as np\n",
"import msgpack\n",
"import queue\n",
"import time\n",
"import zlib\n",
"from tqdm import tqdm\n",
"import subprocess\n",
"from collections import deque\n",
"import matplotlib as plt\n",
"\n",
"def push_socket(local_out_port):\n",
" context = zmq.Context()\n",
" socket = context.socket(zmq.REQ)\n",
" socket.setsockopt(zmq.SNDHWM, 100000)\n",
" socket.setsockopt(zmq.LINGER, 0)\n",
" address = f\"tcp://127.0.0.1:{local_out_port}\"\n",
" socket.bind(address) # Binding to a local port\n",
" print(f\"PUSH socket bound to {address}\")\n",
" return socket\n",
"\n",
"def pull_socket(local_in_port):\n",
" context = zmq.Context()\n",
" socket = context.socket(zmq.REP)\n",
" socket.setsockopt(zmq.RCVHWM, 100000)\n",
" socket.setsockopt(zmq.LINGER, 0)\n",
" address = f\"tcp://127.0.0.1:{local_in_port}\"\n",
" socket.bind(address) # Binding to a different local port\n",
" print(f\"PULL one socket bound to {address}\")\n",
" return socket\n",
"\n",
"\n",
"# Compress image\n",
"def compress_image(image, quality=95):\n",
" # Set the JPEG quality parameter\n",
" encode_param = [int(cv2.IMWRITE_JPEG_QUALITY), quality]\n",
" # Encode the image as a JPEG\n",
" result, encimg = cv2.imencode('.jpg', image, encode_param)\n",
"\n",
" if not result:\n",
" raise Exception(\"Image encoding failed\")\n",
"\n",
" # Decode the encoded image back to an image format\n",
" decimg = cv2.imdecode(encimg, 1)\n",
" return decimg\n",
"# Decompress image\n",
"def decompress_image(encimg):\n",
" image = cv2.imdecode(np.frombuffer(encimg, np.uint8), cv2.IMREAD_COLOR)\n",
" return image\n",
"\n",
"# Global variables\n",
"frames_array=deque(maxlen=2000)\n",
"source_frame = None\n",
"is_manyFace = None\n",
"frameSize = '640x480'\n",
"fps = None\n",
"\n",
"#functions\n",
"def pull_worker(pull_socket):\n",
" global source_frame,is_manyFace,frameSize,fps\n",
" while True:\n",
" try:\n",
" # Receive the JSON with total chunks\n",
" meta_data_json = pull_socket.recv_json()\n",
" #print(meta_data_json)\n",
" total_chunk = meta_data_json['total_chunk']\n",
" # Send acknowledgment for metadata\n",
" pull_socket.send_string(\"ACK\")\n",
" # Receive the array bytes\n",
" source_array_bytes =b''\n",
" for i in range(total_chunk):\n",
" chunk = pull_socket.recv()\n",
" source_array_bytes += chunk\n",
" pull_socket.send_string(f\"ACK {i + 1}/{total_chunk}\")\n",
"\n",
"\n",
" end_message = pull_socket.recv()\n",
" if end_message == b\"END\":\n",
" pull_socket.send_string(\"Final ACK\")\n",
"\n",
" # Deserialize the bytes back to an ndarray\n",
" source_array = np.frombuffer(source_array_bytes, dtype=np.dtype(meta_data_json['dtype_source'])).reshape(meta_data_json['shape_source'])\n",
"\n",
" #plt.imshow(source_array[:, :, ::-1])\n",
" #plt.show()\n",
" #frame_queue.append([\"source\", source_array])\n",
" source_frame = source_array\n",
" is_manyFace = meta_data_json['manyface']\n",
" frames_array.append([\"source\",source_array,is_manyFace])\n",
"\n",
"\n",
" #process_queue.put((\"source\", source_array))\n",
" #break\n",
" except zmq.Again:\n",
" # Sleep briefly to avoid busy-waiting\n",
" time.sleep(0.01)\n",
" except Exception as e:\n",
" print(f\"Error: {e}\")\n",
"def pull_worker_two(pull_socket_two):\n",
"\n",
" while True:\n",
" try:\n",
"\n",
" # Receive the JSON with total chunks\n",
" meta_data_json = pull_socket_two.recv_json()\n",
" #print(meta_data_json)\n",
" total_chunk = meta_data_json['total_chunk']\n",
" # Send acknowledgment for metadata\n",
" pull_socket_two.send_string(\"ACK\")\n",
" # Receive the array bytes\n",
" temp_array_bytes =b''\n",
" for i in range(total_chunk):\n",
" chunk = pull_socket_two.recv()\n",
" temp_array_bytes += chunk\n",
" pull_socket_two.send_string(f\"ACK {i + 1}/{total_chunk}\")\n",
"\n",
"\n",
" end_message = pull_socket_two.recv()\n",
" if end_message == b\"END\":\n",
" pull_socket_two.send_string(\"Final ACK\")\n",
"\n",
" # Deserialize the bytes back to an ndarray\n",
" temp_array = np.frombuffer(temp_array_bytes, dtype=np.dtype(meta_data_json['dtype_temp'])).reshape(meta_data_json['shape_temp'])\n",
" #if source_frame is not None:\n",
" frames_array.append([\"temp\",temp_array])\n",
" print(\"added\",len(frames_array))\n",
" #break\n",
"\n",
"\n",
"\n",
"\n",
" except zmq.Again:\n",
" # Sleep briefly to avoid busy-waiting\n",
" time.sleep(0.01)\n",
" except Exception as e:\n",
" print(f\"Error: {e}\")\n",
"def push_worker(push_socket):\n",
"\n",
" global source_frame,is_manyFace,frameSize,fps\n",
" temp_frm = None\n",
" source_frm = None\n",
" is_manyface = None\n",
" try:\n",
" while True:\n",
" # Get the processed array and metadata from the queue\n",
" if len(frames_array)>0:\n",
" #print(frames_array)\n",
" item = frames_array.popleft()#process_queue.get()\n",
" if item[0]==\"source\":\n",
" source_frm = item[1]\n",
" is_manyface = item[2]\n",
" if item[0]==\"temp\":\n",
" temp_frm = item[1]\n",
" print(\"Recieved\")\n",
"\n",
" if temp_frm is not None and source_frm is not None:\n",
"\n",
" processed_frm =process_frame(get_one_face(source_frm),temp_frm,is_manyface)\n",
"\n",
" face_bytes = processed_frm.tobytes()\n",
" chunk_size = 1024*200\n",
" total_chunk = len(face_bytes) // chunk_size + 1\n",
" metadata ={\n",
"\n",
" 'dtype_source':str(processed_frm.dtype),\n",
" 'shape_source':processed_frm.shape,\n",
" 'size':'640x480',\n",
" 'fps':'60'\n",
" #'shape_temp':temp_frame.shape\n",
" }\n",
" new_metadata = {'total_chunk': total_chunk}\n",
" metadata.update(new_metadata)\n",
" # Send metadata first\n",
" push_socket.send_json(metadata)\n",
" # Wait for acknowledgment for metadata\n",
" ack = push_socket.recv_string()\n",
" with tqdm(total=total_chunk, desc=\"Sending chunks\", unit=\"chunk\") as pbar:\n",
" for i in range(total_chunk):\n",
" chunk = face_bytes[i * chunk_size:(i + 1) * chunk_size]\n",
" # Send the chunk\n",
" push_socket.send(chunk)\n",
" # Wait for acknowledgment after sending each chunk\n",
" ack = push_socket.recv_string()\n",
" pbar.set_postfix_str(f'Chunk {i + 1}/{total_chunk} ack: {ack}')\n",
" pbar.update(1)\n",
"\n",
" # Send a final message to indicate all chunks are sent\n",
" push_socket.send(b\"END\")\n",
" # Wait for the final reply\n",
" final_reply_message = push_socket.recv_string()\n",
" print(f\"Received final reply: {final_reply_message}\")\n",
"\n",
" temp_frm = None\n",
" source_frm = None\n",
" is_manyface = None\n",
"\n",
"\n",
" except Exception as e:\n",
" print (f\"Error in Push Wokr {e}\")\n",
"\n",
"local_in_source = 5555\n",
"local_in_temp = 5556\n",
"local_out_frame = 5557\n",
"\n",
"# Create sockets\n",
"pull_socket_ = pull_socket(local_in_source)\n",
"pull_socket_two = pull_socket(local_in_temp)\n",
"push_socket_ = push_socket(local_out_frame)\n",
"# Run both workers in separate threads\n",
"# Start the pull worker thread\n",
"pull_thread = threading.Thread(target=pull_worker, args=(pull_socket_,))\n",
"pull_thread.start()\n",
"# Start the push worker thread\n",
"pull_thread_two = threading.Thread(target=pull_worker_two, args=(pull_socket_two,))\n",
"pull_thread_two.start()\n",
"# Start the push worker thread\n",
"push_thread = threading.Thread(target=push_worker,args=(push_socket_,))\n",
"push_thread.start()\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "a9JW1rhH3H-T"
},
"source": [
"### [Optional] Open FRP tunnel"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VNDmeHcx3Zrx"
},
"outputs": [],
"source": [
"\n",
"frpc_config = f\"\"\"serverAddr = \"194.113.64.71\"\n",
"serverPort = 7000\n",
"\n",
"[[proxies]]\n",
"name = \"Pull-tcp\"\n",
"type = \"tcp\"\n",
"localIP = \"127.0.0.1\"\n",
"localPort = {local_in_source}\n",
"remotePort = 6000\n",
"\n",
"[[proxies]]\n",
"name = \"Pull-tcp_two\"\n",
"type = \"tcp\"\n",
"localIP = \"127.0.0.1\"\n",
"localPort = {local_in_temp}\n",
"remotePort = 6001\n",
"\n",
"[[proxies]]\n",
"name = \"Push-tcp\"\n",
"type = \"tcp\"\n",
"localIP = \"127.0.0.1\"\n",
"localPort = {local_out_frame}\n",
"remotePort = 6002\"\"\"\n",
"with open('frp_0.59.0_linux_amd64/frpc.toml', 'w') as f:\n",
" f.write(frpc_config)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#Remote tcp\n",
"print(f'tcp://194.113.64.71:6000 ---> {local_in_source}')\n",
"print(f'tcp://194.113.64.71:6001 ---> {local_in_temp}')\n",
"print(f'tcp://194.113.64.71:6002 ---> {local_out_frame}')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ucjrp4fM3mQ4"
},
"outputs": [],
"source": [
"from subprocess import Popen, PIPE\n",
"import time\n",
"ps = Popen('/content/deepfakecollab/Scripts/open_tunnel_frs.sh', stdout=PIPE, stderr=PIPE)\n",
"time.sleep(3)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "SDFT4XXJkVxT",
"outputId": "1738d849-a3e1-403d-95d0-d3a4e4f6cd44"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Opening tunnel\n",
"\u001b[1;34m2024-08-03 20:10:33.967 [I] [sub/root.go:142] start frpc service for config file [frp_0.59.0_linux_amd64/frpc.toml]\n",
"\u001b[0m\u001b[1;34m2024-08-03 20:10:33.967 [I] [client/service.go:294] try to connect to server...\n",
"\u001b[0m\u001b[1;34m2024-08-03 20:10:34.377 [I] [client/service.go:286] [b793872517fab479] login to server success, get run id [b793872517fab479]\n",
"\u001b[0m\u001b[1;34m2024-08-03 20:10:34.377 [I] [proxy/proxy_manager.go:173] [b793872517fab479] proxy added: [Pull-tcp Pull-tcp_two Push-tcp]\n",
"\u001b[0m\u001b[1;34m2024-08-03 20:10:34.514 [I] [client/control.go:168] [b793872517fab479] [Pull-tcp] start proxy success\n",
"\u001b[0m\u001b[1;34m2024-08-03 20:10:34.514 [I] [client/control.go:168] [b793872517fab479] [Pull-tcp_two] start proxy success\n",
"\u001b[0m\u001b[1;34m2024-08-03 20:10:34.514 [I] [client/control.go:168] [b793872517fab479] [Push-tcp] start proxy success\n",
"\u001b[0m^C\n"
]
}
],
"source": [
"!/content/deepfakecollab/Scripts/open_tunnel_frs.sh"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "buCnH_YpxKWa"
},
"source": [
"### [Optional] Open Ngrok tunnel"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"id": "HyY2B5bmxlui"
},
"outputs": [],
"source": [
"from subprocess import Popen, PIPE\n",
"import shlex\n",
"import json\n",
"import time\n",
"\n",
"\n",
"def run_with_pipe(command):\n",
" commands = list(map(shlex.split,command.split(\"|\")))\n",
" ps = Popen(commands[0], stdout=PIPE, stderr=PIPE)\n",
" for command in commands[1:]:\n",
" ps = Popen(command, stdin=ps.stdout, stdout=PIPE, stderr=PIPE)\n",
" return ps.stdout.readlines()\n",
"\n",
"\n",
"def get_tunnel_adresses():\n",
" info = run_with_pipe(\"curl http://localhost:4040/api/tunnels\")\n",
" assert info\n",
"\n",
" info = json.loads(info[0])\n",
" for tunnel in info['tunnels']:\n",
" url = tunnel['public_url']\n",
" port = url.split(':')[-1]\n",
" local_port = tunnel['config']['addr'].split(':')[-1]\n",
" print(f'{url} -> {local_port} [{tunnel[\"name\"]}]')\n",
" if tunnel['name'] == 'input':\n",
" in_addr = url\n",
" elif tunnel['name'] == 'inputtwo':\n",
" in_addrtwo = url\n",
" elif tunnel['name'] == 'output':\n",
" out_addr = url\n",
" else:\n",
" print(f'unknown tunnel: {tunnel[\"name\"]}')\n",
"\n",
" return in_addr,in_addrtwo, out_addr"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"id": "Lnqdi11sxGGR"
},
"outputs": [],
"source": [
"config =\\\n",
"f\"\"\"\n",
"version: 2\n",
"authtoken: {authtoken}\n",
"region: {region}\n",
"console_ui: False\n",
"tunnels:\n",
" input:\n",
" addr: {local_in_source}\n",
" proto: tcp\n",
" inputtwo:\n",
" addr: {local_in_temp}\n",
" proto: tcp\n",
" output:\n",
" addr: {local_out_frame}\n",
" proto: tcp\n",
"\"\"\"\n",
"\n",
"with open('ngrok.conf', 'w') as f:\n",
" f.write(config)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"id": "bQiD7DmexYvy"
},
"outputs": [],
"source": [
"# (Re)Open tunnel\n",
"ps = Popen('/content/deepfakecollab/Scripts/open_tunnel_ngrok.sh', stdout=PIPE, stderr=PIPE)\n",
"time.sleep(3)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "4kC3axJEx5e8",
"outputId": "21867038-e5c4-4038-9876-c41a569b09e7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"tcp://0.tcp.eu.ngrok.io:11640 -> 5556 [inputtwo]\n",
"tcp://4.tcp.eu.ngrok.io:16503 -> 5555 [input]\n",
"tcp://0.tcp.eu.ngrok.io:13155 -> 5557 [output]\n",
"Tunnel opened\n"
]
}
],
"source": [
"# Get tunnel addresses\n",
"try:\n",
" in_addr,in_addr_two, out_addr = get_tunnel_adresses()\n",
" print(\"Tunnel opened\")\n",
"except Exception as e:\n",
" [print(l.decode(), end='') for l in ps.stdout.readlines()]\n",
" print(\"Something went wrong, reopen the tunnel\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "F9Qs1U8gu-l3"
},
"source": [
"### **DeBugging (Ignore)**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "d_gTh_hfbstK"
},
"source": [
"### View Source Image"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "otrLL3NnHCt3",
"outputId": "60a822ea-b55c-4f31-cf19-ddc4390de81f"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"False\n"
]
}
],
"source": [
"print(is_manyFace)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "9Kqc-HRYwa08",
"outputId": "0ff9e514-b029-4df2-c52f-230ddd9dd2a6"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\n"
]
}
],
"source": [
"print(len(frames_array))"
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [
"GYFbuuCdvTaO",
"0rTCd2R1uw7A",
"9n-qESinvM22",
"JmNGrasvwnVX",
"sNHGT2FjjZiK",
"DTbmyO-CK-8b",
"FehUyVNvGGgZ",
"Eb_cbyoaHKrT",
"a9JW1rhH3H-T",
"buCnH_YpxKWa",
"d_gTh_hfbstK"
],
"gpuType": "L4",
"machine_shape": "hm",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 0
}