What is the best way to start a blog post on MongoDB and Java programming? Any idea? Take your time, it also took me some time to figure out. And the answer is: Some really big relational table model of course :-).
Relational Table View
After being infected with some MongoDB nanoprobes and starting with some very small Java sample program on GRIDFS it is time for a bigger Java project using MongoDB.
For these kind of sample projects it is sometimes sufficient to use some fictional code to be able to show how certain things might work. But for this series I wanted to do something more meaningful in a rather small project that has a real purpose for me (and hopefully later on for a few others as well as it is an open-source project ). The following box contains the minimum required details to grasp an understanding of what follows.
Generic Testdata Framework – This is an extension on top of the Robot Framework , which is a test automation framework. The very basic idea is to separate the implementation of the automated testcases completely from the actual tests (input data and expected results). In the current version this can be done using Excel as an input format, but this was always meant to evolve into the possibility to utilise a database to be able to put a web-fronend on top of this. This information should be enough to understand what follows in this (and the forthcoming) posts. More information can be nevertheless found from here . Did I already mention that I strongly believe that this will be the Google of test automation frameworks once it is finished ;-)?
Transformation – From Relational Schema to Document-Oriented Schema
The funny thing is that I did the relational model shown above quite some time back when I had only heard about MongoDB, but had no real clue what it is about. And honestly I always found some excuses to not start any real coding on this as I was considering it quite boring. But while attending the MongoDB class the idea formed in my head that this might be a very good fit here. It is about storing (test) documents and from what I have learned so far that is something where MongoDB shines. So the first step will be to “convert” my nice relational database schema into a document schema to decide how many and which document types (collections) there will be. One thing is for sure: A one-to-one mapping from the tables described above to corresponding documents would be really bad.
Obviously on top-level we can have projects. Then we have definitions of testcases which will result in actual tests using the metadata information from the parameter definition table and the content from the parameter values tables. It is really hard for me – being used to relational database design – to combine any of this information into one document, but I give it a try. My first thought was applying a really “extreme” approach by combining everything into one big document on project-level. But I reconsidered this as as the documents would really become quite complex and potentially also quite huge. What feel quite natural is combining documents on testcase-level, thus putting things from the testcase definition, paramater definition and parameter values tables into one document and linking those to another collection storing the actual project documents. Into those documents then some additional information like the runtime parameters can be put. This would also mean that it is very unlikely that there is a need later on to update both those collections at the same time.
Let’s take a look how the schema would then look like. This can be done best by example (to be honest, I would not know any other way):
db.gtf_projects
{
_id : 'rating_engine',
name : 'Rating Engine'
description : 'This is the test project for the rating engine calculating values ...'
runtime_definitions : {
envId1 : {
selenium-server : 'localhost',
selenium-port : 8988
},
envId2 : {
...
}
}
}
The first thing that can be noticed here is that I sacrificed the relation to a TESTCASE_ID as it would simply not fit in here. I think that is ok, as it is not really needed for runtime parameters. No problem to have them global on project-level. What I do not like here is that the parameter-names used in the different env-sections could easily differ for entries. But probably this a “problem” one has to live with in this kind of flexible schema design. And on the other hand those documents will be written and read by a program in the end. From that point of view I like it, as I can easily retrieve a list of all the runtime parameters for one environment along with the general project information.
Next thing is the definition of one testcase and its variations (actuals tests):
db.gtf_testcases
{
_project_id : 'rating_engine',
name : 'Special SUV Rates',
description : 'Calculating different special rates for SUVs',
param_definitions : {
paramId1 : {
name : 'Horse Power',
desc : 'Used to ...'
},
paramId2 : {
name : 'Color',
desc : 'Used to ...'
},
}
test_definitions : {
testId1 : {
test : 'Description of the specific test',
paramId1 : '110',
paramId2 : 'red'
},
testId2 : { ...
},
}
Did I already mention that this still feels a bit strange. Anyway, one good thing is that those documents are ordered and thus there is no need for any additional “order-column”. It is a bit odd that one will read in all the test definitions in order to get the testcase metadata (parameter names), but on the other hand most likely both things will anyway be needed at the same time in the end. Of course we need a pointer back to the project using the project id. On the other hand I do not mind the ids used for the testcase documents. Therefore I leave it to MongoDB to generate those.
For sure there will be some minor tweaks to this schema, but that’s the one to start with for the implementation.
Starting the Implementation
Sorry, still no MongoDB Java hacking right away. I want to share one consideration I did on this first which is: Using a framework on top of the MongoDB Java driver or not? Looking back at how SQL development has evolved it started (for me) with pure JDBC . Then I was moving over – for a quite short period of time – to Spring JDBC templates . And nowadays it seems to be unthinkable to perform any database access not using JPA . So probably in a bigger project I would use “Spring Data MongoDB” to get shielded from some of the low level programming aspects (Tobias has blogged on this one here ). But for the time being I will take the opportunity to see how things work kind of on the native MongoDB level.
In the last part of this series I just downloaded the MongoDB Java-driver and added it to my project in Eclipse. Of course this is very old-school and therefore I am doing this with Maven this time, especially as my GTF-project is already based on Maven. (Of course this might be still considered semi-old-school nowadays as Gradle is the next cool thing.)
1<dependency> 2 <groupId>org.mongodb</groupId> 3 <artifactId>mongo-java-driver</artifactId> 4 <version>2.9.3</version> 5</dependency>
As a last thing for today I would like to create one project document and inspect it afterwards in the Mongo-shell. I will do so by writing the needed classes and then some kind of integration test (using JUnit) to trigger those classes (as there is not yet any nice GUI to do so).
So here we start. In the following some Java code is shown to make it more convenient to read through this post. Of course everything is also available from the corresponding GitHub-project where things might be more complete and more up-to-date. Finally a short disclaimer: This is my first real (Java) program that is utilising MongoDB (no, it is not my first Java program overall). It should just give an impression how this could be done and hopefully major flaws will be corrected via the comment section :-).
Typically there is some class needed that can create the initial database connection. Not much different here and this is what the following class does. I am trying to keep things relatively short here, therefore I removed the method that can also handle authentication, but it can be checked from GitHub.
1package org.robot.gtf.dblayer.mongodb;
2
3import java.net.ConnectException;
4import java.net.UnknownHostException;
5import com.mongodb.CommandResult;
6import com.mongodb.DB;
7import com.mongodb.DBCollection;
8import com.mongodb.Mongo;
9
10/**
11 * This class provides basic methods for establishing a connection to MongoDB
12 * (with or without authorization) and returning collections.
13 * @author thomas.jaspers
14 */
15public class MongoHandler {
16
17 private DB mongoDb;
18
19 /**
20 * Creates a connection to a MongoDB that does not require authentification.
21 * @param hostName Host running MongoDB
22 * @param port Port of the MongoDB instance
23 * @param dbName Name of the database
24 * @throws UnknownHostException Problem during connection
25 * @throws ConnectException Problem during connection
26 */
27 public void connect(String hostName, int port, String dbName)
28 throws UnknownHostException, ConnectException {
29 Mongo mongo = new Mongo(hostName, port);
30 mongoDb = mongo.getDB(dbName);
31
32 CommandResult lastError = mongoDb.getLastError();
33 if (!lastError.ok()) {
34 throw new ConnectException("Error connecting to MongoDB: " + lastError.getErrorMessage());
35 }
36 }
37
38 /**
39 * Return the MongoDB collection with the specified name.
40 * @param name Name of a MongoDB collection
41 * @return Collection object
42 */
43 public DBCollection getCollection(String name) {
44 return mongoDb.getCollection(name);
45 }
46}
Now we need to reconsider for a moment the basic idea of this. In the end this should be the service-layer for accessing the data from some database. To be able to also use potentially some SQL database later (to not force users into MongoDB no matter how much I like it by now) the following interface must be implemented to abstract from the database-technology. Of course more methods (and probably more interfaces) will follow.
1package org.robot.gtf.dblayer;
2
3import org.robot.gtf.dblayer.to.ProjectTO;
4
5/**
6 * Interface for reading and writing project information.
7 * @author thomas.jaspers
8 */
9public interface ProjectRepository {
10
11 /**
12 * Writes the given project information to the corresponding data store.
13 * @param projectTO A Filled ProjectTO.
14 */
15 void write(ProjectTO projectTO);
16}
Now it becomes a little bit difficult to decide in which order the following classes should be shown to minimise confusion. I decided to show the implementation of the above interface first. Basically here the proper MongoDB collection is accessed using the MongoHandler-class. Then another class is used to create the required JSON-representation from a ProjectTO-class which is again completely unaware of any specific technology that us used. The ProjectTO-class is not shown here, but it just consists of attributes and getters and setters.
1package org.robot.gtf.dblayer.mongodb;
2
3import org.robot.gtf.dblayer.ProjectRepository;
4import org.robot.gtf.dblayer.to.ProjectTO;
5import com.mongodb.BasicDBObject;
6import com.mongodb.DBCollection;
7import com.mongodb.DBObject;
8import com.mongodb.WriteConcern;
9
10/**
11 * Read/write access to the project data using MongoDB as the data store.
12 * @author thomas.jaspers
13 */
14public class MongoProjectRepository implements ProjectRepository {
15
16 private static final String COLLECTION_NAME_PROJECTS = "gtf_projects";
17
18 private MongoHandler handler ;
19
20 /**
21 * Constructor taking a MongoHandler with a ready-made connection.
22 * @param handler Connection to MongoDB
23 */
24 public MongoProjectRepository(MongoHandler handler) {
25 this.handler = handler;
26 }
27
28 @Override
29 public void write(ProjectTO projectTO) {
30 DBCollection collection = handler.getCollection(COLLECTION_NAME_PROJECTS);
31
32 ProjectDocument projectDocument = new ProjectDocument(projectTO);
33 DBObject dbObject = projectDocument.getJsonDocument();
34
35 collection.insert(dbObject, WriteConcern.SAFE);
36 }
37}
Probably the following class is the most interesting one with respect to how to work with MongoDB. It is responsible for transforming a ProjectTO-class into JSON-represenation and later on the JSON-representation (when loading the data again) into a pure ProjectTO. What I find really great is how easy it is to store a list (here a hash map) of a list (another hash map).
1package org.robot.gtf.dblayer.mongodb;
2
3import org.robot.gtf.dblayer.to.ProjectTO;
4import com.mongodb.BasicDBObject;
5import com.mongodb.DBObject;
6
7/**
8 * Represents a MongoDB-Document for projects and is at the the time responsible for mapping from
9 * the corresponding ProjectTO to the JSON-Document and vice versa.
10 * @author thomas.jaspers
11 */
12public class ProjectDocument {
13
14 private static final String DOCUMENT_ATTRIBUTE_ID = "_id";
15
16 private static final String DOCUMENT_ATTRIBUTE_NAME = "name";
17
18 private static final String DOCUMENT_ATTRIBUTE_DESC = "desc";
19
20 private static final String DOCUMENT_ATTRIBUTE_RUNTIME_DEFINITIONS = "runtime_definitions";
21
22 private ProjectTO projectTO;
23
24 /**
25 * Constructor to initialize using a ProjectTO.
26 * @param projectTO ProjectTO
27 */
28 public ProjectDocument(ProjectTO projectTO) {
29 this.projectTO = projectTO;
30 }
31
32 /**
33 * Returns the JSON representation of the contained ProjectTO
34 * @return JSON Document
35 */
36 public DBObject getJsonDocument() {
37 BasicDBObject jsonDoc = new BasicDBObject();
38
39 // Setting basic attributes
40 jsonDoc.put(DOCUMENT_ATTRIBUTE_ID, projectTO.getId());
41 jsonDoc.put(DOCUMENT_ATTRIBUTE_NAME, projectTO.getName());
42 jsonDoc.put(DOCUMENT_ATTRIBUTE_DESC, projectTO.getDescription());
43
44 // Setting the List of environments, which contains a list of parameters
45 BasicDBObject environments = new BasicDBObject();
46 for (String envName : projectTO.getEnvironmentParameter().keySet()) {
47 BasicDBObject envParams = new BasicDBObject();
48 envParams.putAll(projectTO.getEnvironmentParameter().get(envName));
49 environments.put(envName, envParams);
50 }
51
52 jsonDoc.put(DOCUMENT_ATTRIBUTE_RUNTIME_DEFINITIONS, environments);
53 return jsonDoc;
54 }
55
56 public ProjectTO getProjectTO() {
57 return projectTO;
58 }
59
60 public void setProjectTO(ProjectTO projectTO) {
61 this.projectTO = projectTO;
62 }
63}
Finally a test-method is required. This one is creating a ProjectTO (something that later on will be done by some GUI-implementation) and is then using the ProjectRepository (here the Mongo-implementation) to create a new document. So the longest part of this method is creating a complex enough ProjectTO.
1@Test
2public void testProjectCreate() {
3
4 ProjectTO to = new ProjectTO();
5 to.setId("rating_engine");
6 to.setName("Rating Engine");
7 to.setDescription("This is the test project for the rating engine calculating values ...");
8
9 Map<String, Map<String, String>> environments = new HashMap<String, Map<String, String>>();
10 Map<String, String> paramsLocal = new HashMap<String, String>();
11 paramsLocal.put("selenium-server", "localhost");
12 paramsLocal.put("selenium-port", "8867");
13 environments.put("local", paramsLocal);
14
15 Map<String, String> paramsTest = new HashMap<String, String>();
16 paramsTest.put("selenium-server", "test");
17 paramsTest.put("selenium-port", "8892");
18 environments.put("test", paramsTest);
19
20 to.setEnvironmentParameter(environments);
21
22 ProjectRepository rep = new MongoProjectRepository(handler);
23 rep.write(to);
24}
And yes, we have created a new document. As there is only one it can be easily displayed.
> db.gtf_projects.findOne()
{
"_id" : "rating_engine",
"name" : "Rating Engine",
"desc" : "This is the test project for the rating engine calculating values ...",
"runtime_definitions" : {
"test" : {
"selenium-server" : "test",
"selenium-port" : "8892"
},
"local" : {
"selenium-server" : "localhost",
"selenium-port" : "8867"
}
}
}
Comes quite close to the planned document structure. (Ok, I admit that I have tweaked that one a bit after seeing the results ;-).) This is a very first shot, but there are a few things that I would like to sum up:
- The document-approach is really different and at least I had to think twice to get something that looked more or less ok for a document-schema. I was a bit confused by the possibilities one gets by putting document in document in document (kind of).
- It was extremely easy and fast to implement. The whole thing took me roughly 2 hours (please do not tell me that can be seen from the code). Much more time was needed for writing this blog entry. Of course I had some knowledge from writing the first example and of course the – ongoing – MongoDB class.
- This example seems to be a really good fit as the test descriptions are documents.
- The way I stored the hashmap with the runtime parameters to the document seems to result in a bit random order. I need to check still how to fix that.
- Implementing this was really fun as there was no bothering with SQL ;-).
What’s next? Well after writing a document it would be good to have a possibility to read it back in out of our Java application. So that will be then the baseline for the next blog post on this. As usual this is to be continued :-).
The MongoDB class series
Part 1 – MongoDB: First Contact
Part 2 – MongoDB: Second Round
Part 3 – MongoDB: Close Encounters of the Third Kind
Part 4 – MongoDB: I am Number Four
Part 5 – MongoDB: The Fith Element
Part 6 – MongoDB: The Sixth Sense
Part 7 – MongoDB: Tutorial Overview and Ref-Card
Java Supplemental Series
Part 1 – MongoDB: Supplemental – GRIDFS Example in Java
Part 2 – MongoDB: Supplemental – A complete Java Project – Part 1
Part 3 – MongoDB: Supplemental – A complete Java Project – Part 2
Was this post helpful?
More articles
fromThomas Jaspers
22.2.2019 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Thomas Jaspers
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.