Hi there,
I have a question on the setup of the development environment for flink. Is it possible to update the big JAR file (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink apparently uses for deployment, with a single module (flink-runtime)? Recompiling all of the modules takes several minutes on my laptop, while compiling only the module is a matter of seconds. My knowledge of maven is pretty limited, so I compile flink with > mvn package -Dmaven.test.skip=true This process is really slowing down my development cycle. I would appreciate your help :) Best regards, Niklas -- PhD Student / Research Assistant INET, TU Berlin Room 4.029 Marchstr 23 10587 Berlin Tel: +49 30 314 78752 |
Hi Niklas,
if you only want to update individual submodules you can do: cd flink-runtime // build this submodule mvn -DskipTests clean install // will put the new flink-runtime jar into the local Maven repository cd ../flink-dist mvn -DskipTests clean install // will fetch all required dependencies from the local Maven repository without rebuilding them. You need to have run mvn -DskipTests clean install before to have all required dependencies in the local repository. Cheers, Fabian 2015-09-04 11:46 GMT+02:00 Niklas Semmler <[hidden email]>: > Hi there, > > I have a question on the setup of the development environment for flink. > > Is it possible to update the big JAR file > (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink > apparently uses for deployment, with a single module (flink-runtime)? > > Recompiling all of the modules takes several minutes on my laptop, while > compiling only the module is a matter of seconds. > > My knowledge of maven is pretty limited, so I compile flink with > > > mvn package -Dmaven.test.skip=true > > This process is really slowing down my development cycle. I would > appreciate your help :) > > Best regards, > Niklas > > -- > PhD Student / Research Assistant > INET, TU Berlin > Room 4.029 > Marchstr 23 > 10587 Berlin > Tel: +49 30 314 78752 > |
One more remark on this:
You need to be a little bit more careful. This approach works only, if the initially changed module has no other dependent modules. For example, flink-clients also depends on flink-runtime and thus, you *might* need to rebuild flink-clients, too, before building flink-dist. However, only if you did API changes in flink-runtime. The basic message is, you need to understand the dependency tree. If you did API changes, you need to rebuild all modules that depend on the changed module. -Matthias On 09/04/2015 12:24 PM, Fabian Hueske wrote: > Hi Niklas, > > if you only want to update individual submodules you can do: > > cd flink-runtime // build this submodule > mvn -DskipTests clean install // will put the new flink-runtime jar into > the local Maven repository > cd ../flink-dist > mvn -DskipTests clean install // will fetch all required dependencies from > the local Maven repository without rebuilding them. > > You need to have run mvn -DskipTests clean install before to have all > required dependencies in the local repository. > > Cheers, Fabian > > 2015-09-04 11:46 GMT+02:00 Niklas Semmler <[hidden email]>: > >> Hi there, >> >> I have a question on the setup of the development environment for flink. >> >> Is it possible to update the big JAR file >> (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink >> apparently uses for deployment, with a single module (flink-runtime)? >> >> Recompiling all of the modules takes several minutes on my laptop, while >> compiling only the module is a matter of seconds. >> >> My knowledge of maven is pretty limited, so I compile flink with >> >>> mvn package -Dmaven.test.skip=true >> >> This process is really slowing down my development cycle. I would >> appreciate your help :) >> >> Best regards, >> Niklas >> >> -- >> PhD Student / Research Assistant >> INET, TU Berlin >> Room 4.029 >> Marchstr 23 >> 10587 Berlin >> Tel: +49 30 314 78752 >> > |
Hello Fabian, Hello Matthias,
Thank you very much! I am not making API changes at the moment, so I should be fine with this approach. Unfortunately the command "mvn -DskipTests clean install" in the main folder didn't work for me (it hanged in the middle of the operation), so I am using a more brute force method. I compile the flink-runtime jar file just as Fabian described and then use it to overwrite the content in the "lib/flink-dist-0.9-SNAPSHOT.jar". It is not really a clean path, but this method has reduced my compile time down to 1 minute, which is a great improvement :) Best, Niklas On 04.09.2015 12:59, Matthias J. Sax wrote: > One more remark on this: > > You need to be a little bit more careful. This approach works only, if > the initially changed module has no other dependent modules. > > For example, flink-clients also depends on flink-runtime and thus, you > *might* need to rebuild flink-clients, too, before building flink-dist. > However, only if you did API changes in flink-runtime. > > The basic message is, you need to understand the dependency tree. If you > did API changes, you need to rebuild all modules that depend on the > changed module. > > > -Matthias > > On 09/04/2015 12:24 PM, Fabian Hueske wrote: >> Hi Niklas, >> >> if you only want to update individual submodules you can do: >> >> cd flink-runtime // build this submodule >> mvn -DskipTests clean install // will put the new flink-runtime jar into >> the local Maven repository >> cd ../flink-dist >> mvn -DskipTests clean install // will fetch all required dependencies from >> the local Maven repository without rebuilding them. >> >> You need to have run mvn -DskipTests clean install before to have all >> required dependencies in the local repository. >> >> Cheers, Fabian >> >> 2015-09-04 11:46 GMT+02:00 Niklas Semmler <[hidden email]>: >> >>> Hi there, >>> >>> I have a question on the setup of the development environment for flink. >>> >>> Is it possible to update the big JAR file >>> (flink-dist/target/.../lib/flink-dist-0.9-SNAPSHOT.jar), which flink >>> apparently uses for deployment, with a single module (flink-runtime)? >>> >>> Recompiling all of the modules takes several minutes on my laptop, while >>> compiling only the module is a matter of seconds. >>> >>> My knowledge of maven is pretty limited, so I compile flink with >>> >>>> mvn package -Dmaven.test.skip=true >>> >>> This process is really slowing down my development cycle. I would >>> appreciate your help :) >>> >>> Best regards, >>> Niklas >>> >>> -- >>> PhD Student / Research Assistant >>> INET, TU Berlin >>> Room 4.029 >>> Marchstr 23 >>> 10587 Berlin >>> Tel: +49 30 314 78752 >>> >> > -- PhD Student / Research Assistant INET, TU Berlin Room 4.029 Marchstr 23 10587 Berlin Tel: +49 30 314 78752 |
Free forum by Nabble | Edit this page |