Part 7a – Configuring Crawl4Neo

Let’s get some boilerplate work out of the way. These steps are necessary for a real Spring application. Fortunately, once this is complete we won’t have to visit this stuff very often.

Packages

This package structure is a little closer to what I would use for a commercial project. Since, our project has taken a big leap forward in complexity now it a good time to start organizing our code.

config

Configuration information, while minimal with Spring Boot, is still necessary. To me it’s just the dirty facts of life and I like to keep it separate from the ‘real’ code. If we were to grow this into a full, commercial web application, then there would be quite a bit of configuration. It’s just a good habit to keep a config package to hold it all.

data

This is where we’ll place all the entities we are going to keep in Neo4J. I put the repositories in this package as well for this project. This might change overtime, but it’s clean.

services

These are the workhorse classes, where stuff really happens. When a class is configured as a Spring Service, it’s best practice to suffix the name with ‘Service’. These are beans we can ‘autowire’ into our application as needed.

Configuration

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.scrumbucket</groupId>
    <artifactId>crawler4neo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.3.3.RELEASE</version>
    </parent>

    <properties>
        <java.version>1.8</java.version>
        <maven.compiler.target>1.8</maven.compiler.target>
        <maven.compiler.source>1.8</maven.compiler.source>
        <powermock.version>1.6.1</powermock.version>
        <junit.version>4.11</junit.version>
        <jsoup.version>1.7.1</jsoup.version>
        <commons-lang3.version>3.0</commons-lang3.version>
        <commons-io.version>2.4</commons-io.version>
        <mockito-all.version>1.10.8</mockito-all.version>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>${junit.version}</version>
        </dependency>
        <dependency>
            <groupId>org.jsoup</groupId>
            <artifactId>jsoup</artifactId>
            <version>${jsoup.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>${commons-lang3.version}</version>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-all</artifactId>
            <version>${mockito-all.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.powermock</groupId>
            <artifactId>powermock-module-junit4</artifactId>
            <version>${powermock.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.powermock</groupId>
            <artifactId>powermock-api-mockito</artifactId>
            <version>${powermock.version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-neo4j</artifactId>
        </dependency>
        <dependency>
            <groupId>org.hibernate</groupId>
            <artifactId>hibernate-validator</artifactId>
        </dependency>
    </dependencies>

    <repositories>
        <repository>
            <id>spring-releases</id>
            <name>Spring Releases</name>
            <url>https://repo.spring.io/libs-release</url>
        </repository>
        <repository>
            <id>neo4j</id>
            <name>Neo4j</name>
            <url>http://m2.neo4j.org/</url>
        </repository>
    </repositories>

</project>

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

<groupId>org.scrumbucket</groupId>

<artifactId>crawler4neo</artifactId>

<version>1.0-SNAPSHOT</version>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-parent</artifactId>

<version>1.3.3.RELEASE</version>

</parent>

<java.version>1.8</java.version>

<maven.compiler.target>1.8</maven.compiler.target>

<maven.compiler.source>1.8</maven.compiler.source>

<powermock.version>1.6.1</powermock.version>

<junit.version>4.11</junit.version>

<jsoup.version>1.7.1</jsoup.version>

<commons-lang3.version>3.0</commons-lang3.version>

<commons-io.version>2.4</commons-io.version>

<mockito-all.version>1.10.8</mockito-all.version>

</properties>

<build>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-maven-plugin</artifactId>

</plugin>

</plugins>

</build>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>${junit.version}</version>

</dependency>

<groupId>org.jsoup</groupId>

<artifactId>jsoup</artifactId>

<version>${jsoup.version}</version>

</dependency>

<groupId>org.apache.commons</groupId>

<artifactId>commons-lang3</artifactId>

<version>${commons-lang3.version}</version>

</dependency>

<groupId>org.mockito</groupId>

<artifactId>mockito-all</artifactId>

<version>${mockito-all.version}</version>

</dependency>

<groupId>org.powermock</groupId>

<artifactId>powermock-module-junit4</artifactId>

<version>${powermock.version}</version>

</dependency>

<groupId>org.powermock</groupId>

<artifactId>powermock-api-mockito</artifactId>

<version>${powermock.version}</version>

</dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter</artifactId>

</dependency>

<groupId>org.springframework.data</groupId>

<artifactId>spring-data-neo4j</artifactId>

</dependency>

<groupId>org.hibernate</groupId>

<artifactId>hibernate-validator</artifactId>

</dependency>

</dependencies>

<id>spring-releases</id>

<name>Spring Releases</name>

<url>https://repo.spring.io/libs-release</url>

</repository>

</repository>

</repositories>

</project>

Lines 11-15 allow us to pull dependency information directly from Spring. They have gone to the work of identifying which versions of what jars play nicely together. Frankly, this parent pom is what makes Spring Boot so attractive.

Lines 31-34 install the Spring Boot plug in. It will help us compile and run our application with minimal effort.

Lines 72-83 list the Spring Framework modules we want in our project. For our purposes we only need to list Spring Boot and Spring Data Neo4j. Spring Boot looks at those jars and figures out what sort of application we are building. It then brings in other jars as needed and wires up a default configuration.

CrawlerConfig.java

package org.scrumbucket.crawler4neo.config;

import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.factory.GraphDatabaseFactory;
import org.scrumbucket.crawler4neo.Crawler4Neo;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.neo4j.config.EnableNeo4jRepositories;
import org.springframework.data.neo4j.config.Neo4jConfiguration;

@Configuration
@EnableNeo4jRepositories(basePackageClasses = Crawler4Neo.class)
public class CrawlerConfig extends Neo4jConfiguration {

    CrawlerConfig() {
        setBasePackage("org.scrumbucket.crawler4neo");
    }

    @Bean
    GraphDatabaseService graphDatabaseService() {
        return new GraphDatabaseFactory().newEmbeddedDatabase("crawler4neo.db");
    }


}

package org.scrumbucket.crawler4neo.config;

import org.neo4j.graphdb.GraphDatabaseService;

import org.neo4j.graphdb.factory.GraphDatabaseFactory;

import org.scrumbucket.crawler4neo.Crawler4Neo;

import org.springframework.context.annotation.Bean;

import org.springframework.context.annotation.Configuration;

import org.springframework.data.neo4j.config.EnableNeo4jRepositories;

import org.springframework.data.neo4j.config.Neo4jConfiguration;

@Configuration

@EnableNeo4jRepositories(basePackageClasses = Crawler4Neo.class)

public class CrawlerConfig extends Neo4jConfiguration {

CrawlerConfig() {

setBasePackage("org.scrumbucket.crawler4neo");

}

@Bean

GraphDatabaseService graphDatabaseService() {

return new GraphDatabaseFactory().newEmbeddedDatabase("crawler4neo.db");

}

@Configuration tells Spring that this file provides configuration information to the system.

@EnableNeo4jRepositories does just was it says. We need to include a basePackageClasses option so Spring will scan for relevant classes. If you leave out basePackageClasses you’ll get the message:

org.springframework.beans.factory.NoSuchBeanDefinitionException: 
No qualifying bean of type [org.scrumbucket.crawler4neo.data.PageRepository] found for dependency: 
expected at least 1 bean which qualifies as autowire candidate for this dependency.

org.springframework.beans.factory.NoSuchBeanDefinitionException:

No qualifying bean of type [org.scrumbucket.crawler4neo.data.PageRepository] found for dependency:

expected at least 1 bean which qualifies as autowire candidate for this dependency.

Basically, Spring won’t see that we have things like Repository beans. So when it tries to wire up our Service that uses the repository, it can’t resolve the dependency.

Alternatively, you could code it this way:

@EnableNeo4jRepositories(basePackages = "org.scrumbucket.crawler4neo")

1	@EnableNeo4jRepositories(basePackages = "org.scrumbucket.crawler4neo")

I found using ‘basePackages’ is error prone. Specifying the package as text means you won’t get a compile error when you refactor your code (which I do relentlessly). ‘basePackageClasses’ simply uses the class you specify to find the base package to start scanning. BTW, I just use a class at the highest level and be done with it. Why specify several package names only to forget one? After all, package scanning only happens at boot time.

Line 16 is an anomaly that the Spring Data Neo4j team got rid of in Version 4. I’m not sure why they required it, but if you leave it out you’ll get this exception when you try to save any data:

Caused by: org.springframework.data.mapping.model.MappingException: Unknown persistent entity org.scrumbucket.crawler4neo.data.Page

1	Caused by: org.springframework.data.mapping.model.MappingException: Unknown persistent entity org.scrumbucket.crawler4neo.data.Page

(We’ll convert our project to version 4 after I understand it better.)

Line 21 is where we give our database a file name. Everything Neo4j needs to save goes in one file – I love it.

This covers our configuration stage. We can now move on to the application.