file - java.lang.NoSuchMethodError in Flink

Question

I trying to read the a file using :

final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
DataSet<String> line = env.readTextFile("file:///pathtofile/myfile.txt");

I get following error:

java.lang.NoSuchMethodError: org.apache.flink.api.common.io.DelimitedInputFormat: method <init>(Lorg/apache/flink/core/fs/Path;)V not found

I'm using flink version 1.3.2, java version "1.8.0_91"

Answer 1

There is a conflict with dependencies. Apache Flink loads many classes by default into its classpath.

Please read this article https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/debugging_classloading.html the last section

Resolving Dependency Conflicts with Flink using the maven-shade-plugin

Apache Flink loads many classes by default into its classpath. If a user uses a different version of a library that Flink is using, often?

IllegalAccessExceptions

?or?

NoSuchMethodError

?are the result.

So, I suggest to play with your pom.xml and use maven-shade-plugin and add correct relocation, as we have in example

<relocation>
  <pattern>org.codehaus.plexus.util</pattern>
  <shadedPattern>org.shaded.plexus.util</shadedPattern>
  <excludes>
    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
  </excludes>
</relocation>