Hi Guys,
this is more of a discussion point than a question.
Over the last 10 years I have pioneered many new ideas. One of them is SeETL. This idea is going to change the future of ETL development.
The question is "are ETL developers on MSFT platform interested in what we have done"?
Feel free to ask questions or comment on this thread about what we have done. I think this as good a place as any to discuss SeETL and how it can be used in conjunction with SSIS.
These two videos will give you an idea of what we have done. I will explain in a few words below. But there is a LOT of information available about what we have done on our web site. Some people will love us, some people will hate us. Those who will hate us
are the ones who build ETL for a living and do not want faster, better, cheaper ways of building ETL to become widely known. There are plenty of those guys around. Luckily they do not control budgets.
http://www.youtube.com/watch?v=3_wXs-vjZNk&feature=plcp
http://www.youtube.com/watch?v=E0PaMJIA7II&feature=plcp
Basically we have cut the cost of ETL development in about half even when a vendor ETL tool is used. I know that sounds like a "tall story" but it is true.
My background is I have been in IT 30 years building large batch systems...and ETL is just one more large batch system. I have been in BI for 21 years now. I built my first ETL tool in 1995 and it was used in many accounts to give me a competitive advantage
on selling deals. So I know what I am talking about.
Starting in 2002 we migrated all our ETL knowledge into SeETL. It has developed over the last 10 years. It gives us a competitive advantage in ETL development to the tune of about 50%. It started out when I was talking to Ralph Kimball who suggested that "if
you are so smart write an article about all the features are good ETL should have in it and I will publish it." That article, which never did get published, formed the initial design specs for SeETL.
SeETL allows us to completely prototype the ETL subsystem prior to actually using a vendor ETL tool. In our clients what we have been doing is building the ETL subsystems in SeETL, getting everything to work, getting the data models settled....doing all the
testing on the prototype....and once the client is happy THEN migrate the ETL to the clients specified ETL tool. We have built a suite of tools that massively help us build data warehouses at our clients.
This is a client we did where we migrated the SeETL ETL to Informatica. http://www.youtube.com/watch?v=jBJWe9FxRxs&feature=plcp
We have ways and means to migrate our ETL to DataStage and Informatica very quickly. This all works on any standard database. DB2, Oracle, Sybase IQ, Netezza, MySQL are all proven platforms. But we are an MSDN and we develop on MSFT.
We have offered IBM and Informatica the opportunity to hire us to write an interface that will generate mappings for their tools. Up to them if they want to accept it.
I have contacted some old friends at MSFT to make the same offer to MSFT for SSIS...but I guess here is as good a place as any to mention that too.
The basic concept is very simple.
Forget about GUIs and build ETL directly from the mapping workbook using vb to read the workbook and generate what ever is necessary.
We have an older C++ engine but we can now also generate SQL directly. That means we are somewhat competitive to SSIS because we can do most of what SSIS can do. However, clients use SSIS for many more reasons than just getting the ETL done. There are also
issues of support etc. So we see SeETL happily co-habiting with ETL tools...some people are using SeETL in production but most are using it only in development.
SeETL covers much more than just the ETL today. Anyone who takes a deeper look into the reports we provide out of the box will see that.
I am now in the process of making the whole idea of "write mappings and other directives in a spreadsheet and generate what is needed" into a widely accepted idea. I fully realise that I am going to be called "crazy" for this idea. I was also called "crazy"
in 93-5 when I was talking about star schemas. But this idea is going to transform the world of BI. Just like star schemas did.
I very much own the idea of putting all the mapping definitions into a workbook and generating what is needed out of the workbook. This idea was invented in 2004 so there is more "prior art" than anyone can poke a stick at. We have quite a few big clients who
have witnessed what we have been able to do. But as a small firm talking to people there is great resistance to this idea as per normal.....people who disbelieve new ideas are very committed to their disbelief...rather like sailors who said the world was flat.
So it is time to get this idea "out there" and widely accepted and adopted. And what better way than to offer some useful "free ware" and let people guide how the product evolves. The usual "free ware" standards apply. The software is free and the support is
paid at normal support rates (which are lower than MSFT in any case).
What we are doing is making BI more accessible to more companies by lowing the cost of developing BI. This is good for everyone except people who make money out of coding ETL. The ideas I am bringing forward will put a lot of those guys out of work, just like
computers put a lot of book keepers out of work. One caveat. You can't do this if you are in the german speaking region. We have an exclusive reseller in the german speaking region so you have to talk to them first.
SeETL for prototyping and design time work will remain free. There are lots of very useful reports and you can build your complete prototype. It will be a matter of your honour that you do not deploy the prototype into production. The only time we look for
SeETL software fees is deployment to production. If a client develops in SeETL and deploys in SSIS and does not call us for support? Hooray for them. Good luck to them. (We actually make our real money out of selling data models not SeETL.)
In production we are looking to bring the price of ETL down as SeETL is more widely accepted. If SeETL is not widely accepted we will continue to sell it for its current price point of EUR20K to EUR50K to deploy into production for one company. A rounding error.
So it is really up to "the best and brightest" to decide for themselves if they want to use SeETL in production or just use it in development and deploy SSIS into production.
So...that is the proposal. The innovative idea that GUIs slow us down when building ETL, Data Models, etc, and that creating the input for such in an excel workbook and generating everything needed is a good idea. I know it is a good idea because at the end
of a project our clients usually get the ETL in a vendor ETL tool and the data model in a data modelling tool. The whole end product looks "normal" and they switch over to "normal" support. A few clients use our tools for everything and they retain their advantage
by doing that.
The point being? We get our clients to the end point faster and cheaper than anyone else.
So....Feel free to discuss this idea on this thread....good...bad...or otherwise. I am very interested in comments. I am interested to see if there are some guys here who are the "best and brightest" who would like to try out what it is we have done and evaluate
it for themselves.
I am very much into "put the idea out there and see what happens." So far, not much. But that is exactly what selling star schemas was like in 93-95...by 2000 everyone had "discovered" star schemas thanks to Ralf Kimball and his books. We will see the same
again in this tool I think.
Best regards
Peter Nolan