Serving static web resources in an OSGi environment

After having the persistence layer, I was thinking how to serve static resources in an OSGi based web environment. I definitely do not want to create WABs with web.xml files. This is I have come up so far:

In case we would like to offer static resources from a bundle we add the following capability to it one or more times:
Provide-Capability: static.resources
Possible attributes of the capability:
  • path: Path in the bundle where from we would like to export resources
  • filePattern: Pattern of the files that can be served. E.g.: *.gif
  • recurse: Export files from sub-directories or not. Default is true.
  • scope: Only export files from the bundle or from the imported packages as well. Possible values: local / wiring. Default is local
  • mimeConfig: Location of an optional properties file inside the bundle where file extension – metatype values are paired. E.g.: gif=image/gif
  • someAttributeKey: Optional attributes that will be listed in the OSGi service properties (see below)
A full example:
Provide-Capability: static.resources;path=/pathinbundle/;filePattern=*;recurse=true;scope=wiring;someAttributeKey=someValue

An extender picks up every bundle that contains the “static.resources” capability and registers a StaticResourceProvider OSGi service for each capability. The service properties come from the capability attributes. The functions of the StaticResourceProvider interface:

  • listFiles(String folder): Listing files and sub-directories inside the specified folder
  • getResourceAsStream(String path)
  • getMimeType(String path)
The StaticResourceProvider OSGi services can be used by the component called StaticResourceServlet. The configuration options of the component:
  • staticResourceProviders.target: OSGi filter that addresses one or more static resource provider service
  • acceptRanges: Whether range queries are accepted or not (HTTP 1.1)
  • dirAllowed: If directory listing is allowed or not
  • styleSheet: Location of a CSS static resource that could decorate directory listing
  • welcomeFileNames: Comma separeated list of welcome file names. If directory url is provided, it will be redirected to the welcome file if that exists.
  • gzFileRegex: If a file name with path matches with this optional regular expression, it will be sent unpacked, via a compressed stream if available (HTTP 1.1)
  • etags: Whether etags are allowed or not (HTTP 1.1)
  • maxCacheSize: The maximum total size of the cache or 0 for no cache.
  • maxCachedFileSize: The maximum size of a file to cache
  • maxCachedFiles: The maximum number of files to cache
  • cacheControl: If set, all static content will have this value set as the cache-controlheader
  • otherServiceProps: String array of key-value pairs that should be put into the properties of the registered OSGi service. E.g.: Putting whiteboard pattern specific properties.

The configuration properties of StaticResourceServlet component can be familiar from DefaultServlet of Jetty.

Downloading WSDL files for offline use

Another blog post from the past.

We had the problem that one of the developer worked on the service layer of a project and the other wanted to use the functions of it. The service layer was exposed to the internet as web services. As the function signatures of the service layer had not been finalized yet there was the question how to re-generate the client side code on the other side each times the api had been changed.

It was pretty clear that as using maven for compiling the source the ws-import maven plugin should be used. However an url of the wsdl file must be provided for that plugin and this is where the problems came into the picture. The wsdl contained several xsd:import xml tags which made it pretty hard to download always the wsdl files with the xsd files it imports by hand. Also the value of schemalocation attribute had to be always changed to the location of the downloaded xsd files.

Luckily it was not hard to write a code snippet that downloads a wsdl (which is an xsd as well) from an url and if it imports other xsd files it downloads them as well. Also the code snippet changes the schemalocation in the downloaded wsdl files.

The code snippet is the following:

 
 package hu.everit.utils.xsd.downloader;

import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import java.util.Iterator;
import java.util.Map;

import javax.xml.namespace.NamespaceContext;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.transform.Result;
import javax.xml.transform.Source;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerConfigurationException;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;

import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;

public class XsdDownloader {

	public static class XsdNameSpaceContext implements NamespaceContext {

		private final Map<String, String> nameSpaceUrisByPrefixes;

		public XsdNameSpaceContext() {
			nameSpaceUrisByPrefixes = new HashMap<String, String>();
			nameSpaceUrisByPrefixes.put("xsd",
					"http://www.w3.org/2001/XMLSchema");
		}

		public String getNamespaceURI(final String prefix) {
			return nameSpaceUrisByPrefixes.get(prefix);
		}

		public String getPrefix(final String namespaceURI) {
			// TODO Auto-generated method stub
			return null;
		}

		public Iterator getPrefixes(final String namespaceURI) {
			// TODO Auto-generated method stub
			return null;
		}

	}

	/**
	 * @param args
	 */
	public static void main(final String[] args) {
		if (args.length != 2) {
			System.out.println("Only two parameters: 1-url 2-downloadPrefix");
			return;
		}
		String xsdUrl = args[0];
		String filePrefix = args[1];
		XsdDownloader xsdDownloader = new XsdDownloader();
		xsdDownloader.setDownloadPrefix(filePrefix);
		try {
			xsdDownloader.downloadXsdRecurse(xsdUrl);
		} catch (TransformerConfigurationException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (ParserConfigurationException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (SAXException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		} catch (TransformerException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	Map<String, String> fileNamesByprocessedUrls =
			new HashMap<String, String>();

	private String downloadPrefix;

	private void downloadXsdRecurse(final String xsdUrl) throws IOException,
			ParserConfigurationException, SAXException, TransformerException {

		String outputFileName = downloadPrefix;
		if (fileNamesByprocessedUrls.size() > 0) {
			outputFileName =
					outputFileName + "." + fileNamesByprocessedUrls.size();
		}
		outputFileName = outputFileName + ".xsd";
		fileNamesByprocessedUrls.put(xsdUrl, outputFileName);

		DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
		dbf.setNamespaceAware(true);
		DocumentBuilder db = dbf.newDocumentBuilder();
		Document doc = db.parse(xsdUrl);

		processElementRecurse(doc.getDocumentElement());

		File outputFile = new File(outputFileName);
		TransformerFactory trf = TransformerFactory.newInstance();
		Transformer tr = trf.newTransformer();
		Source source = new DOMSource(doc);
		Result result = new StreamResult(outputFile);
		tr.transform(source, result);
	}

	private void processElementRecurse(final Element node) throws IOException,
			ParserConfigurationException, SAXException, TransformerException {
		NodeList nl = node.getChildNodes();
		for (int i = 0, n = nl.getLength(); i < n; i++) {
			Node childNode = nl.item(i);
			if (childNode instanceof Element) {
				Element childElement = (Element) childNode;
				if ("http://www.w3.org/2001/XMLSchema".equals(childElement
						.getNamespaceURI())
						&& childElement.getLocalName().equals("import")) {
					System.out.println("foundElement");
					String schLoc = childElement.getAttribute("schemaLocation");
					if (!fileNamesByprocessedUrls.containsKey(schLoc)) {
						downloadXsdRecurse(schLoc);
						String newLoc = fileNamesByprocessedUrls.get(schLoc);
						if (newLoc != null) {
							childElement.setAttribute("schemaLocation", newLoc);
						}
					} else {
						String newLoc = fileNamesByprocessedUrls.get(schLoc);
						childElement.setAttribute("schemaLocation", newLoc);
					}
				} else if ("http://schemas.xmlsoap.org/wsdl/"
						.equals(childElement.getNamespaceURI())
						&& childElement.getLocalName().equals("import")) {
					System.out.println("foundWsdlElement");
					String schLoc = childElement.getAttribute("location");
					if (!fileNamesByprocessedUrls.containsKey(schLoc)) {
						downloadXsdRecurse(schLoc);
						String newLoc = fileNamesByprocessedUrls.get(schLoc);
						if (newLoc != null) {
							childElement.setAttribute("location", newLoc);
						}
					} else {
						String newLoc = fileNamesByprocessedUrls.get(schLoc);
						childElement.setAttribute("location", newLoc);
					}
				} else {
					processElementRecurse(childElement);
				}
			}
		}
	}

	public void setDownloadPrefix(final String downloadPrefix) {
		this.downloadPrefix = downloadPrefix;
	}

}
The main function of the class above accepts two parameters. The first one is the url of the wsdl file and the second is the prefix of the file that will be the beginning of the downloaded files. The downloaded files will be named as PREFIX.xsd, PREFIX.1.xsd, PREFIX.2.xsd, … The first one is the main one that the specified url contained, the others are the imported. The code snippet also takes care of the cyclic imports so the files that were downloaded ones already will not be downloaded again.
The mavenized project with the compiled executable jar is available here. To run it enter the following command: java -jar everit-xsd-downloader-1.0.jar URL FILEPREFIX 

Everit Liquibase modules on maven-central

Recently, I decided to leave JPA and try something new concerning persistence. I evaluated many technologies and found Liquibase and QueryDSL to be the best for our needs. Both of them have their unique advantages.

Liquibase is perfect for maintaining database schemas. It is also possible to follow version changes when a deployment is performed and also to generate upgrade SQL scripts.

QueryDSL could work as a replacement of JPA Criteria API for us. JPA helped us to start a completely new way of programming. We could place our entities into separate modules and write utility functions that generate only a snippet from the complete SQL statement. However, we reached the limitations of JPA Criteria API soon. After I started to use QueryDSL, I felt that I superseded those limitations.

To use these technologies, we needed new tools and modules. We decided to use OSGi as the base of our framework, therefore, all our technologies must be modularized. By modularization, I do not only mean to extend the MANIFEST.MF file of the JAR files, but also to extend the functionality of our technologies to make it possible to use them within the concept of modularization. The following tools were implemented recently:

  • liquibase-bundle: I extended the functionality of Liquibase to be able to import database schema snippets from different bundles. The snippets can be imported transitively. I also enhanced the standard OSGi support of Liquibase a little bit. With the change, extensions can be deployed as fragment bundles. I would like to contribute to the Liquibase project with the code so that I do not have to maintain the changes.
  • liquibase-component: A simple OSGi Declarative Services component that uses the functionality of Liquibase-bundle to process changelog files. The component can be configured. One option is to upgrade the database, the other one is to dump the changes to SQL scripts so the db admin can perform the upgrade later. I am not sure if this component will have many future versions. It might happen that the code of the component will be integrated into the liquibase-datasource.
  • liquibase-datasource: A configurable DataSource component. It needs a DataSource to access the database and also a database schema expression that points to the liquibase changelog file. After the database migration is finished, the component registers a new DataSource OSGi service.
  • osgi-lqmg: Liquibase-QueryDSL-Metadata-Generator can be used to generate QueryDSL metadata classes directly from Liquibase changelog files. This tool uses the OSGi concept too, so instead of specifying the exact location of the changelog file, a Bundle-Capability must be defined.
  • lqmg-maven-plugin: A maven plugin that uses osgi-lqmg to generate Liquibase Metadata classes. The plugin deploys the dependencies into an embedded OSGi container and uses them to find the Liquibase changelogs.

With the tools above, it is possible to use Liquibase and QueryDSL together easily within the OSGi world. I hope that this will become a common solution in the future.

Single Sign-Out with CAS in .NET

The old website of Everit will be closed soon. Therefore I am starting to backup the blog posts I wrote years ago and that still have several visitors. Let this be the first one:

There are several tutorials in the CAS wiki how to do SSO (Single Sign On) inside a .NET web application and CAS. There is a CAS client as well that is currently in 1.0 pre state. I tried it but did not work well and as there is not much time I had to find a workaround that handles Single Sign Out as well using FormsAuthentication.

Based on the tutorial available in the CAS wiki I created my own. The difference is that I created a static Dictionary that stores the tickets and the belonging sessions in weak references. Periodically it is checked if a session is not used anymore and that ticket is cleared from the this dictionary (not to have memory leak). When there is a sign out message from CAS the changed code gets out the entry and sets a flag to say next time somebody visits the site the user is logged out. Also the page loader of the master site checks the Dictionary in each request if a logout occured and if yes it drops to the logout page The following steps are necessary to have the solution:

  1. Be sure that both IIS and the CAS server are accessed via https (they do not work with simple http) and they both can see each other based on their domain.
  2. Create a new form called CASLogin.aspx and add a new Label onto it. This label will have the name Label1.
  3. Add the attribute ValidateRequest=”false” to the <%@ Page… element in CASLogin.aspx
  4. Add the following code into CASLogin.aspx.cs (change the url of the cas server):
     using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Web;
    using System.Web.UI;
    using System.Web.UI.WebControls;
    using System.IO;
    using System.Net;
    using System.Xml;
    using System.Web.Security;
    using System.Net.Security;
    using System.Text.RegularExpressions;
    
    namespace WebApplication2.Account
    {
        public partial class CASLogin : System.Web.UI.Page
        {
            // Local specific CAS host
            private const string CASHOST = "https://localhost:8181/cas/";
    
            protected void Page_Load(object sender, EventArgs e)
            {
                String logoutRequest = Request.Params.Get("logoutRequest");
                if (logoutRequest != null)
                {
                    Regex regex = new Regex("<.*\\:?SessionIndex\\>(.*)\\<\\/.*\\:?SessionIndex>");
                    Match match = regex.Match(logoutRequest);
                    if (match != null && match.Success)
                    {
                        string signOutTicket = match.Groups[1].Value;
                        try {
                        CASUtil.CASTicket casTicket = CASUtil.sessionTickets[signOutTicket];
                        casTicket.LogoutHappened = true;
                        
                        } catch (KeyNotFoundException ex) {
    
                            // Do not care about it
                        }
                        return;
                    }
                }
                // Look for the "ticket=" after the "?" in the URL
                string tkt = Request.QueryString["ticket"];
    
                // This page is the CAS service=, but discard any query string residue
                string service = Request.Url.GetLeftPart(UriPartial.Path);
    
                // First time through there is no ticket=, so redirect to CAS login
                if (tkt == null || tkt.Length == 0)
                {
                    string redir = CASHOST + "login?" +
                      "service=" + service;
                    Response.Redirect(redir);
                    return;
                }
    
                // Second time (back from CAS) there is a ticket= to validate
                string validateurl = CASHOST + "serviceValidate?" +
                  "ticket=" + tkt + "&" +
                  "service=" + service;
    
                //Change SSL checks so that all checks pass
                ServicePointManager.ServerCertificateValidationCallback =
                    new RemoteCertificateValidationCallback(
                        delegate
                        { return true; }
                    );
       
                StreamReader Reader = new StreamReader(new WebClient().OpenRead(validateurl));
                string resp = Reader.ReadToEnd();
                // I like to have the text in memory for debugging rather than parsing the stream
    
                // Some boilerplate to set up the parse.
                NameTable nt = new NameTable();
                XmlNamespaceManager nsmgr = new XmlNamespaceManager(nt);
                XmlParserContext context = new XmlParserContext(null, nsmgr, null, XmlSpace.None);
                XmlTextReader reader = new XmlTextReader(resp, XmlNodeType.Element, context);
    
                string netid = null;
    
                // A very dumb use of XML. Just scan for the "user". If it isn't there, its an error.
                while (reader.Read())
                {
                    if (reader.IsStartElement())
                    {
                        string tag = reader.LocalName;
                        if (tag == "user")
                            netid = reader.ReadString();
                    }
                }
                // if you want to parse the proxy chain, just add the logic above
                reader.Close();
                // If there was a problem, leave the message on the screen. Otherwise, return to original page.
                if (netid == null)
                {
                    Label1.Text = "CAS returned to this application, "
                        + "but then refused to validate your identity.";
                }
                else
                {
    
                    HttpContext.Current.Session.Add("CASTicket", tkt);
                    CASUtil.CASTicket casTicket = new CASUtil.CASTicket(HttpContext.Current.Session);
                    CASUtil.sessionTickets.Add(tkt, casTicket);
                    Label1.Text = "Welcome " + netid;
                    FormsAuthentication.RedirectFromLoginPage(netid, false); // set netid in ASP.NET blocks
                }
            }
        }
    }
    
    
  5. Create a new class with the name CASUtil and add the following code into CASUtil.cs:
     using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Web;
    
    namespace WebApplication2
    {
        public class CASUtil
        {
    
            public static Dictionary<String, CASTicket> sessionTickets;
    
            private static System.Timers.Timer sessionTicketsChecker = new System.Timers.Timer();
    
            static CASUtil()
            {
                sessionTickets = new Dictionary<String, CASTicket>();
                sessionTicketsChecker = new System.Timers.Timer(2000);
                sessionTicketsChecker.Elapsed += new System.Timers.ElapsedEventHandler(CASUtil.checkSessionTickets);
                sessionTicketsChecker.Enabled = true;
            }
    
            private static void checkSessionTickets(object sender, System.Timers.ElapsedEventArgs e)
            {
                Dictionary<string, CASTicket>.KeyCollection kCol = sessionTickets.Keys;
                foreach (String ticket in kCol)
                {
                    if (!sessionTickets[ticket].HttpSessionStateWR.IsAlive)
                    {
                        sessionTickets.Remove(ticket);
                    }
                }
            }
    
    
            public class CASTicket
            {
                public CASTicket(System.Web.SessionState.HttpSessionState session)
                {
                    this.httpSessionStateWR = new WeakReference(session);
                }
                private WeakReference httpSessionStateWR;
    
                private bool logoutHappened = false;
    
                public WeakReference HttpSessionStateWR
                {
                    get { return httpSessionStateWR; }
                    set { httpSessionStateWR = value; }
                }
    
                public bool LogoutHappened
                {
                    get { return logoutHappened; }
                    set { logoutHappened = value; }
                }
            }
        }
    }
    
  6. Set the following in web.config:
     <authentication mode="Forms">
          <!-- <forms loginUrl="~/Account/Login.aspx" timeout="2880" />-->
          <forms name="casauth" loginUrl="~/Account/CASLogin.aspx" />
        </authentication>
        <authorization>
          <deny users="?" />
        </authorization>
    
  7. Add the following to the beginning of web.config inside <system.web>:
     <httpRuntime requestValidationMode="2.0" />
    
    
  8. Add the following to Default.aspx.cs:
     using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Web;
    using System.Web.UI;
    using System.Web.UI.WebControls;
    using System.Web.Security;
    
    namespace WebApplication2
    {
        public partial class _Default : System.Web.UI.Page
        {
            protected void Page_Load(object sender, EventArgs e)
            {
                try
                {
                    string ticket = (string)Context.Session["CASTicket"];
                    if (ticket != null)
                    {
                        CASUtil.CASTicket casTicket = CASUtil.sessionTickets[ticket];
                        if (casTicket.LogoutHappened)
                        {
                            FormsAuthentication.SignOut();
                            FormsAuthentication.RedirectToLoginPage();
                        }
                    }
                }
                catch (KeyNotFoundException ex)
                {
                    //Do nothing
                }
                
    
            }
        }
    }
    

Step 3 and 7 are very important with the entry ValidateRequest=”false” in CASLogin.aspx otherwise you will get a http 500 error when the CAS server sends the SAML sign out message. This is because if validate request is turned on ASP.NET does not allow any form parameter that contains possible xml elements. You can find more information about it here.

There might be other and better solutions but I am not a .NET developer. I guess there is a better place to do the flag check than the page load function of the default page. Also with the static variables the solution does not work via a cluster. Also I guessed that a user has to login to see any page. Otherwise the session should be invalidated not by dropping to the logout page but somehow else.

Everit Development Tools on maven-central

The development tools and components of Everit, which make it much easier to develop OSGi based applications together with maven, are finally available on maven-central. With the new releases it is now possible to:

  • Use any of the existing OSGi environments (felix and equinox are supported at the moment)
  • Create a custom OSGi environment and run the developed application on it
  • Replace maven dependencies in the OSGi container without restarting it
  • Change code and test them by dropping the projects onto an always-on-top window (no full compilation is necessary)
  • Run the tests during the integration-test phase of a maven build (useful on Continuous Integration servers)

To get more information about the usage of the tools, visit our GitHub repositories.

YAJSW on maven-central

As YAJSW is used heavily in eosgi-maven-plugin, we uploaded it to maven central with the grouId “org.rzo.yajsw”. This is one of the last steps of releasing eosgi-maven-plugin 2.0.0.

Everit on maven-central

In the last couple of weeks / months we worked hard to publish the next major release of our projects . As a result, many of our artifacts were uploaded to maven-central: http://search.maven.org/#search%7Cga%7C1%7Ceverit

Follow

Get every new post delivered to your Inbox.