(here presented in Maven syntax, but the same dependencies apply to other build tools (Gradle, SBT, etc.) as well. When setting up a project manually, you need to add the following dependencies for the Java/Scala API The user application dependencies explicitly do not include the Flink DataStream APIs and runtime dependencies,īecause those are already part of Flink’s Core Dependencies.Įvery Flink application needs as the bare minimum the API dependencies, to develop against.
INTELLIJ JAR FOR MAPREDUCE CODE
The user application is typically packaged into an application jar, which contains the application code and the required
The User Application Dependencies are all connectors, formats, or libraries that a specific user application needs. To keep the default classpath small and avoid dependency clashes. In fact, we try to keep the core dependencies as slim as possible Number of dependencies and classes in the classpath by default. The Flink Core Dependencies do not contain any connectors or libraries (CEP, SQL, ML, etc.) in order to avoid having an excessive Which contains the classes like String and List. Think of these dependencies as similar to Java’s core library ( rt.jar, charsets.jar, etc.), Part of the basic Flink container images. These core classes and dependencies are packaged in the flink-dist jar. The set of all these classes and dependencies forms the core of Flink’s runtime and must be present when a Flink Runtime library must be available as well.Īs with most systems that run user-defined applications, there are two broad categories of dependencies and libraries in Flink:įlink Core Dependencies: Flink itself consists of a set of classes and dependencies that are needed to run the system, for exampleĬoordination, networking, checkpoints, failover, APIs, operations (such as windowing), resource management, etc. When running Flink applications (either in a distributed deployment, or in the IDE for testing), the Flink Many applications depend in addition on certain connector libraries (like Kafka, Cassandra, etc.). At the bare minimum, the application depends Appendix: Template for building a Jar with DependenciesĮvery Flink application depends on a set of Flink libraries.Adding Connector and Library Dependencies.Setting up a Project: Basic Dependencies.Flink Core and Application Dependencies.Hadoop MapReduce compatibility with Flink.Conversions between Table and DataStream.Conversions between PyFlink Table and Pandas DataFrame.
0 kommentar(er)
