Question / Help Building ffmpeg and OBS to use NVENC on Linux Mint.

[JRFrogman]

I am building a system to use OBS on Linux. It is a brand new install of Linux Mint V18.3 Mate 64Bit. The uname -a is : Linux Liberty01 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux. I am trying to take advantage of the MSI GeForce GTX 1050 hardware.

I have followed a number of forum threads on how to accelerate FFMPEG by building a version to take advantage of NVENC. Many trial and errors with old info that required patching the code. As I understand it now, ffmpeg has everything it needs to use NVENC. You just need to configure it (--enable-nvenc) and then build it. I am using a minimal configuration to keep things simple. I will add more libraries later when I get this working. Here is what I did to build ffmpeg:

## First I installed the tools needed for everything.
sudo apt-get install yasm build-essential automake autoconf libtool pkg-config cmake git-core checkinstall

## I did install all the libraries called for by the various recipes but I probably won't use them right away.
sudo apt-get install libx11-dev libgl1-mesa-dev libvlc-dev libpulse-dev libxcomposite-dev \
libxinerama-dev libv4l-dev libudev-dev libfreetype6-dev \
libfontconfig-dev qtbase5-dev libqt5x11extras5-dev libx264-dev \
libxcb-xinerama0-dev libxcb-shm0-dev libjack-jackd2-dev libcurl4-openssl-dev

## I created a directory to put this project in.
mkdir Development
cd Development

## I cloned a copy of the FFMPEG source code.
git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg
cd ffmpeg/

## I didn't want the latest untested stuff so I specifically called for the latest release. I am not a git expert, so this step may have been unnecessary.
git checkout tags/n3.4.2

## I made a build directory, jumped in, and configured ffmpeg with what I figured was minimal.
mkdir ffmpeg_build
cd ffmpeg_build
../ffmpeg/configure --prefix=/usr --extra-version=2 --toolchain=hardened \
--libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu \
--enable-shared \
--enable-gpl \
--enable-nonfree \
--enable-nvenc

## I then built ffmpeg and installed it. Yeah, it is a little 4 core i5.
make -j 4
sudo checkinstall --pkgname=FFmpeg

## Everything worked great. I listed the encoders and Nvida was there.
## So I just used your OBS build recipe.

cd ..
git clone --recursive https://github.com/jp9000/obs-studio.git
cd obs-studio
mkdir build && cd build
cmake -DUNIX_STRUCTURE=1 -DCMAKE_INSTALL_PREFIX=/usr ..
make -j4

This is where everything keeps getting stuck.

[ 32%] Building C object libobs/CMakeFiles/libobs.dir/audio-monitoring/pulse/pulseaudio-output.c.o
[ 32%] Linking C shared library libobs.so
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libswscale.a(swscale.o): relocation R_X86_64_PC32 against symbol `ff_M24A' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
libobs/CMakeFiles/libobs.dir/build.make:2002: recipe for target 'libobs/libobs.so.0' failed
make[2]: *** [libobs/libobs.so.0] Error 1
CMakeFiles/Makefile2:451: recipe for target 'libobs/CMakeFiles/libobs.dir/all' failed
make[1]: *** [libobs/CMakeFiles/libobs.dir/all] Error 2
Makefile:149: recipe for target 'all' failed
make: *** [all] Error 2

I did try and add --enable-pic to the ffmpeg build and -DCMAKE_C_FLAGS="-fPIC" to the OBS build and I still got the same error. It obviously looks like a linker error but where did it start? I was a good C\C++ coder back in the 80s-90s then switched to Java. Most of this would take a lot of digging for me. I was just hoping that this was a common error and someone would know what it was right away. Help??

 

[JRFrogman]

SO....It seems that there really isn't any support for this project. It seems like this would be a common problem among those trying to build this project and yet no one has a clue?? Little to no community help for anything beyond encoding settings etc...??? Great!!!

 

[Tuna]

Perhaps because the primary issue seems to be with the FFMPEG library and how it is build instead of OBS. Luckily on my Debian system it wasn't even necessary to compile FFMPEG at all. All I had to do was to install NVIDIA encoder libraries and I was set.

info: ---------------------------------
info: NVENC supported
info: ---------------------------------

But back to the original problem - it indicates that libobs wants to link a static FFMPEG library. That seems weird. Usually it would link to the shared library by default (I would assume). And so it does for me:

$ ldd /usr/local/lib/libobs.so | grep swscale
    libswscale.so.4 => /usr/lib/x86_64-linux-gnu/libswscale.so.4 (0x00007fddff440000

So maybe you have only installed the static version of the FFMPEG library you have compiled or it requires a rerun of Cmake to pick up the shared ones.

 

[JRFrogman]

I have always had the Nvidia graphics drivers installed first thing in any new OS install. So they were there through all the other iterations of this. As for Nvidia libs, I went through many other iterations where I was downloading libs and running patches on FFMPEG. Those recipes worked and I was able to see the -encoders. I just was not able to --enable-shared. Then I found out that the latest version of FFMPEG already had all that was needed to use NVENC. I just had to get the source and --enable-nvenc. It worked and I could still see the Nvidia encoders. It was a breeze and there never seemed to be a problem with building FFMPEG, just conflicts when using --enable-shared.

This is a new Mint install and I didn't install FFMPEG from the repositories. So I can't imagine where the static libs would come from. When I rebuilt FFMPEG, I used --enable-shared.

Everything worked fine on the FFMPEG build. I never had a problem with it. The first time around I did use FFMPEG from the repository and I did see some libs that had an older date so I figured I was not replacing them during the install. Just to be sure, I reinstalled Mint and never touched the repositories for FFMPEG or OBS. I also used checkinstall. My assumption was to make sure the system knew where all the libs were. But I am not great with understanding how the package managers work.

As for the libobs.so, it isn't in /usr/local/lib. I did find it in /usr/lib/x86_64-linux-gnu/compiz. libswscale.so.4 is not in its list of dependencies. I am thinking it probably isn't the lib we are looking for because it is in the compiz directory and I never got OBS to finish linking. libobs.so is part of the OBS build right?. It does have a few nvidia lib dependencies but I think that is for graphics acceleration to draw the desktop.

I know I am getting long winded here but I am just trying to give a complete picture and I could very well be wrong in some of my assumptions. So the next question seems to be what FFMPEG libs is OBS looking for during linking, where should they be located, and does the system need to be told where they are?

 

[JRFrogman]

Looking at the errors, it seems that the linker is trying to link the object libobs.so against /usr/lib//usr/lib/gcc/x86_64-linux-gnu/libswscale.a(swscale.o) and there is an incompatibility. libswscale.a is built in a way that the OBS project doesn't work with. So I am guessing the OBS project knows where to find the libraries, at least in this one case. The question is which project needs to change. I am thinking the FFMPEG project is specifically built to use the Nvidia hardware (--enable-nvenc) and we also need --enable-shared????

But the linker error seems like it is some kind of problem of linking static with dynamic libs??? I guess I need to check on the libswscale.a and see???? I am guessing that the FFMPEG project built the libswscale.a the way it is needed (with --enable-shared)??? So there needs to be a way to tell the OBS project to also make a compatible object????

 

[JRFrogman]

So libswscale.a is a static library and built in the FFMPEG project??? But I used --enable-shared on that project. Why did this happen??? Am I missing something????

 

[JRFrogman]

Ok. There is a shared object library in there:


-rw-r--r-- 1 root root 6351990 Mar 15 15:14 /usr/lib/x86_64-linux-gnu/libswscale.a
lrwxrwxrwx 1 root root 28 Feb 13 2017 /usr/lib/x86_64-linux-gnu/libswscale-ffmpeg.so.3 -> libswscale-ffmpeg.so.3.1.101
-rw-r--r-- 1 root root 550904 Feb 13 2017 /usr/lib/x86_64-linux-gnu/libswscale-ffmpeg.so.3.1.101
lrwxrwxrwx 1 root root 21 Mar 15 15:14 /usr/lib/x86_64-linux-gnu/libswscale.so -> libswscale.so.4.8.100
lrwxrwxrwx 1 root root 21 Mar 15 15:14 /usr/lib/x86_64-linux-gnu/libswscale.so.4 -> libswscale.so.4.8.100
-rwxr-xr-x 1 root root 550896 Mar 15 15:14 /usr/lib/x86_64-linux-gnu/libswscale.so.4.8.100

But there is also a static (libswscale.a) library in there. Why is the OBS project trying to link with libswscale.a instead of libswscale.so?????

 

[JRFrogman]

Ok, still not getting anywhere and not getting any HELP.

I tried many iterations on the build based on what I am seeing here. OBS seems to want to link to the static library libswscale.a. So why not disable the static stuff in FFMPEG??? I found other examples of people building FFMPEG with --disable-static (for OBS Studio). I did this and then no libswscale.a but OBS doesn't like it. I tried just moving libswscale.a so that only libswscale.so was available. OBS complained again that it didn't know how to create libswscale.a. So it wants libswscale.a but complains that it isn't compiled with fPIC. I did build FFMPEG with --enable-pic AGAIN and checked it as best I could with objdump and there seems to be a lot of relocation tables there. Still got the same problem when building OBS.

On researching this issue I found a lot of people complaining about FFMPEG being a bitch to build. No where near OBS. You can configure the build on FFMPEG easily. The wrapper that goes around FFMPEG and depends heavily on FFMPEG, not so much.

I have wasted more time that anyone should on such a project without getting any help. I am starting to wonder if all the actual developers left this project and all that is left are the people who can only pick at the settings page. Or was there a fork and all the smart people when with the other version and I am just trying to resurrect a dead fork??? These are the things you should ask yourself before putting too much time into this project. You may just have to lump it and go with something on Windoze.

 

[pkv]

go to discord to get help or on irc, that's where most support is done.
Remember that this is an open source project, not a company so all helpers on irc/discord/forum are volunteers !
there: https://obsproject.com/help
links to discord and irc

 

[Osiris]

Your commands work fine on Ubuntu 17.04, obs and libobs link to the shared objects fine here.
Also it's probably best to completely rerun cmake (empty out the build folder) and check where it detects ffmpeg and if that is the correct location.

 

[JRFrogman]

pkv,

Yes, I understand that it is open source. I have a lot of experience with it since I use Linux almost exclusively. I was an IT manager for many years and almost our entire infrastructure was Linux and open source. When I had difficulties, I could usually find a post on that exact same problem and the solution worked great. If things changed, you could usually find a more recent post with another working solution. When I found a solution for a problem that had not been seen and had the time to know WHY it worked, I would post that. I didn't have time to sit in the IRC forums to help out but that is just the way I work. I also tried the IRC channel for this project and got nothing. But I like written online documentation anyway.

Most projects are very well supported by volunteers because most of them have seen all the problems that can arise. That is the whole point of open source; so you can use what has already been done and you don't have to reinvent the wheel. You also want to learn from the mistakes that others have been through so you don't have to waste time on the same problem. Usually when you can't get to a solution, we all work to define a problem space and then figure out the problem, multiple minds are better than one, another reason I like open source. When you get nothing, either people don't have a clue or they don't care. You are basically on your own and it defeats the purpose of open source. Then you just have a FREE software project instead of a COMMUNITY project. I believe open source is about knowledgeable people sharing knowledge. When open source is running the way it was intended, it works waaaaay BETTER than any software put out by corporations.

With all the NVidia cards out there and people wanting to leverage that hardware, you do find a lot of people who are doing this same thing going about 5 to 6 years back I think. I saw many examples of people wanting to use NVENC on Nvidia hardware and they all came to the same conclusion, rebuild it from the source. There have also been quite a few examples of the same linking problem that I ran into but no one touched it.

At this rate I don't know if I will have any more time to waste on this problem now. I have been at this for quite some time and I am building this system for a friend of mine. He needs this system and I may have to move on with another solution. But if I can solve it quickly or recreate the problem in a VM, I will try and get to the bottom of this and post it someplace. This problem seems to affect more than just this project but my guess is it is something really easy, like a compiler switch, and people more familiar than I with GNU build tools should see at first glance.

 

[JRFrogman]

Osiris,

I am old school and used to build these packages before package managers. We always "make clean" or something similar after each iteration of a problem. My version of Mint uses Ubuntu 16.04 so I may have another machine I can try it on. I am assuming that you have tried it on 16.04 already?? I do have VMs but I don't know if I can recreate the problem there.

I would prefers not to use Ubuntu 17.04 because I am building this system for a friend who teaches and lectures on The Constitution. He knows very little about computers so I just want to set up a system that he can use long term, turn key, everything set up for him. That is one reason I did not install the short term version 17.04 on his machine, even though there were many posts that said NVENC was supported right out of the box. There must be a change in 16.04 and 17.04 that makes rebuilding this easier.

My guess is that this is a common problem depending on how a library is created. There is probably a linker or compiler switch that is needed and no one here knows it. But it may be known in other groups, probably the GNU build tools groups.

 

[pkv]

i understand your pain. Can be tricky. I lost a lot of time when i started building obs on linux and mac.
My guess is you have some apps which installed ffmpeg libs and that this conflicts with those you compiled.
I experienced it so i decided to install ffmpeg libs in my home (as in ffmpeg own wiki tuto for ubuntu).
Here's what I do for example:

cd ffmpeg
PATH="$HOME/bin:$PATH" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure \
--prefix="$HOME/ffmpeg_build" \
--pkg-config-flags="--static" \
--extra-cflags="-I$HOME/ffmpeg_build/include" \
--extra-ldflags="-L$HOME/ffmpeg_build/lib" \
--bindir="$HOME/bin" \
--enable-shared \
--enable-gpl \
--enable-libass \
--enable-libfdk-aac \
--enable-libfreetype \
--enable-libopus \
--enable-libtheora \
--enable-libvorbis \
--enable-libx264 \
--enable-nvenc \
--enable-nonfree

make -j8
sudo make install
ldconfig ===> did you ldconfig ?

For obs compilation, for cmake:
cmake -DUNIX_STRUCTURE=1 -DFFmpegPath=$HOME/ffmpeg_build/ -DCMAKE_INSTALL_PREFIX=/usr ..

Report and for further troubleshooting do check on discord ! will be simpler.

 

[JRFrogman]

pkv,

This is a new OS install and nothing there other than what the installation puts there. I definitely didn't have any ffmpeg libs until I built from source. As you can probably see from earlier posts, I am trying to remove as many variables as I can, so I am not enabling any more libs than are necessary in the ffmpeg config.

Last Friday I built a couple of VMs, one with the version of Mint I am using and one with Ubuntu 17.04. Since I do not have the resources to add an Nvidia card into a VM, I built the project on the Mint VM without --enable-nvenc on the ffmpeg project. It just compiled and linked without a problem. Of course there were no Nvidia encoders present but it proved the minimal recipe used to build the project worked. I immediately tried it on the system I am building but without the --enable-nvenc. Still no joy. I am thinking it has something to do with the presence of the Nvidia libs. I am using Nvidia Driver Version 384.111 on Linux Mint.

I figure the problem exists somewhere with ffmpeg and the Nvidia libs. Since ffmpeg always compiles and links with --enable-nvenc but OBS does not, I do not believe there to be a problem with ffmpeg. So I am stumped until I feel like getting familiar with this version of GNU C/C++ compilers and linkers. Since no one on the OBS project seems to know what that linker error means, I should just walk away if Ubuntu 17.04/FFMPEG/OBS has Nvidia support right out of the box and then 18.04 has a good chance of also supporting Nvidia. But I do believe this problem with plague the OBS project until someone figures out what is happening. But I don't want to work on this alone so I am out.