Faceswap Forum

Normally when converting VR videos, you convert both faces at the same time, the left angle of the face and the right angle as the VR clip is two videos side by side from slightly different angles. Since I've updated (it probably wasnt the most recent one, I usually only update once every couple weeks as long as things are working). Converting two faces per frame used to work but now doesn't. Now it only will convert a section of the left face for a couple seconds, then instantly switch over to a section of the right face for a couple sections, then back to the left face, and etc etc. Like it wants to convert both sides at the same time, but now it can only convert one face per frame. How do I fix this to make it back to two faces per frame?

If you include the alignments file with all faces identified it should work as you describe with it converting multiple faces. I have not done this in VR videos, but have tested it with reflections where it works as I'd expect.

Reflections is a perfect example. It used to work in reflections for me too just like it used to work in VR, but now it doesn't. And I dont think this is alignment/extraction related because if my video is 10 seconds x 30 fps x 2 faces (cause VR), you would expect that I would have extracted 600 faces, and when you check my extraction output, you will find 600 faces. So clearly this alignment got the 600 thing right, but when convert that 10 second clip. it will on convert essentially 300 faces, because it flickers between the two, only 1 face per frame. Its almost like in the past several updates, some invisible, inaccessible tick box was activated where convert would only convert one face per frame.

Vr videos are giving me problems so I tried see if I can replicate it with a different scenario, two completely unrelated videos clips playing side by side in a single mp4 file, like a vr video or a mirror reflection. I converted it and had the same exact issue, only one face at a time. My faceswap is up to date and there were no problems with installation or anything. If you could give any more feed back bryan, that would be really appreciated. I've tried playing with lots of settings in extraction and conversion.

What happens if you open up the video in the manual tool? Does it show both faces in each frame there?

I replicated the scenario a third time, this attempt was to deepfake some pictures with a bunch of faces inside of them (of the same person) in each picture, and got the same results. Instead of every single face in that picture being converted, only one would randomly be converted. I'm surprised no one else has came across this yet.

To answer your question torz, when I open up the vr video in the manual tool, I see the frame with both faces in the big preview, and both faces of that green extract box and purple bounding box, alongside both having the light blue mesh and landmarks, But theres a large green number on the faces. "1" and "0" and it seems kind of randomly chosen. for the first 10 frames the left face will be "1" and the right will be "0" then for a couple frames right will be "1" and left will be "0". The switching of these numbers does seem to correlate with which face is being converted per frame in my conversion.
Other things:
-The slider under the main preview shows all "300 faces" and when I click on one of the thumbnails below, two will be selected at a time, likely the 2 faces in each frame.

• Clicking thumbnail filter "has multiple faces" shows all "300 frames".
• I dont know what any of these things do or mean, but bounding box editor is on aligner FAN and normalization is on Hist. Both faces have boxes.
  -In location editor, both faces have boxes, but those numbers show up again.
• Landmark points editor boxes and numbers both faces,
  -mask editor is on components mask type and brush size of 10. Both faces have a red mask on them
  Does this tell you what you need? Or what I need to do?

Way more information than I need (which makes a pleasant change )

I'll check what's happening when I have a second.

Just done a very quick test using a model trained from similar faces but not those in the sources for the swap, hence the swap is a little poor.
BUT, every face swaps, so it can be done.
Original vid is a UK TV add for car insurance with the same actor doing two characters over blue screen and then superimosed.

I wonder If I need to uninstall and reinstall everything. Maybe the problem is somehow entirely on my end
Edit: Update, I deleted faceswap and reinstalled it, and I also found that I had some nvidia game ready drivers that need updating. unfortunately neither of these solved my problem, And I even tried extracting a new clip to get a brand new alignment file to convert with. The only think left I can think of would be uninstalling literally everything like tensorflow, cuda, etc.

I'm no expert but ......
Have you got a standalone version of CUDA installed ? If so uninstall it as FS comes with it's own version, uninstall FS, search for a folder folder called MiniConda3 (which on Windows will be in your user/username folder), delete that and then reinstall FS. Although I would have expected training arrors before trying to convert if the above is the case.

I also found the Studio Nvidia drivers work better than the Game Ready Drivers.

I doubt this is a Cuda thing....

Can you hit "Generate" on the command you convert with (and only swaps 1 face) and post the output.

Thanks Djang anfd torzdf for the reply and follow up. I still have no solution to this issue.
.
Djang I did delete the entire faceswap file and re install it. And yeah, no real errors in training.
Torzdf: If I understand what you are saying, you want me to convert a two faced video, but instead of converting, hit generate and copy and paste the code to you:

C:\Users\USERNAME\MiniConda3\envs\faceswap\python.exe C:\Users\USERNAME\faceswap\faceswap.py convert -i "D:/Nueral Network programs/ProjectPersonA/VideoClips/DF 4_Trim.mp4" -o "D:/Nueral Network programs/ProjectPersonA " -al "D:/Nueral Network programs/ProjectPersonA/VideoClips/DF 4_Trim_alignments.fsa" -m "D:/Nueral Network programs/ProjectPersonA/DFLH128 /MD" -c avg-color -M extended -sc sharpen -w ffmpeg -osc 100 -l 0.4 -j 0 -L INFO

Another update: I tried opening another project with a different trainer model, different input A and B faces, and different videos and alignments. I know multiple faces jpegs and vr videos worked on them before but they are also not working like my current project -so basically this seems to be affecting me on every model and project and probably on faceswap
.
Any possible additional convert info: Main tab: No reference video, AVG color, extended mask, sharpen, ffmpeg
Convert plugins: Clip and preserve paper selected, manual balance and match hist on install default, mask guassian default, scaling sharpen on gaussian, writer is mp4 libx264 settings default quality 26 preset veryfast
.
Edit: Tried deleteing faceswap and miniconda 3 and reinstalling them and it still did not work.
If we run out of solutions to fix this issue, is there anyway I could get a link to an older version of faceswap? like one from a month ago?

The same exact thing is happening to me, and it used to work before the last update. If I open up tools -> manual, everything looks good there. Every frame has a 0 and a 1. However, when I convert, the 0 gets ignored and only the 1 converts.

If you can share a video + alignment file where this problem occurs (Private Message is fine) then I may have time to have a look at the weekend.

You will already know this because its sent to you, but just for everyone else to see. I have sent you 5 different before videos, alignment files, and afters videos(conversions). I would really like to have this issue solved, So I'm trying to send you the max amount of data possible. I don't know the core issue here, but I feel like If I changed some of the video types and extraction settings it may yield new results for you. So the videos include:

1. A simulated VR video, that has two slightly different faces of the same clip playing at the same time, extended mask, 30fps.

2. A regular clip of 3 different people talking, 30 fps, extended mask, ANDno faces were sorted and removed so all three faces should be converting

3. That same clip of the three only this time its shorter, It's 60fps, it was extracted and converted in a VGG obstructed mask, ANDonce again no sorting or removing of face alignments so all three should be converted

4. That same clip again, its 30fps and extended mask, BUT THIS TIME facees were sorted, and removed faces was used on the old man, so now all his alignments should be gone and it should only convert 2 people.

5. A picture Deepfake. The previous 4 were all videos so this time I made a collage in paint, a bunch of different pictures of the same person all in one jpeg file. VGG Obstructed on. The end result should be all the faces in this multi-face collage should be converted.

-Naturally All 5 of these are examples of it not working for me. In all 5 examples, only one face was converted when multiple faces should.
-The file just named "alignments" is the picture deepfake alignment file. Also I included the before and after face removal alignments for the face removal video
-For some reason on triple face long 30fps, it prefers to stay on oldman 95% if the time. The same is true with triple face short 60fps VGG Extended. The simulated VR video best shows the phenomenon where it constantly switches between faces. Regardless, it should be on all faces, not just oldman or switching back and forth.
-Ignore the deepfake quality, this model I used which previously used to work with multiple face conversions was not trained for these three people in the clip.

Message sent with a video, alignments file and generated output

Thanks for these. I definitely won't have a chance to look at this before the weekend, but I will look at the issue as soon as I can

Anyone figured this out? I have the exact same issue, old version swapped all faces fine but now that I’ve updated it will choose one frame or the other (on VR) and randomly swap that frames face but not the other.

This should be fixed in latest commit. Please update.

Thank you. So far it seems to be working for me now. Ive tried a multi-face jpeg collage deepfake and a VR deepfake and it seems to work. Check your DMs