database - Best practices for huge volumes of data load/unload? -

-


database - Best practices for huge volumes of data load/unload? -

my question applies etl scenarios, transformation performed outside of database (completely). if extract, transform, , load huge volumes of info (20+ 1000000 records or more) , databases involved : oracle , mssql server, best way to:

effectively read source database : there way avoid querying on network? have heard things direct path extract method/ mass unload method - i'm quite not sure how work, presume need dump file of sorts kind of non-network based info read/import? effectively write transformed info target database?: should consider apache hadoop? help me start transformation , parallely load info destination database? - faster say, oracle's mass load utility? if not,, there way remote invoke mass load utlities on oracle/mssql server?

appreciate thoughts/suggestions.

i utilize db's mass load facilities this. remote command of mass loads sysadmin issue; there way this.

this means first 2 stages of etl , application generates right file format mass load facility, , lastly stage invoke mass loading.

database oracle design-patterns architecture

Comments

Post a Comment

Popular posts from this blog

Using Android Volley to post an array to PHP -

-
Using Android Volley to post an array to PHP - this how post values volley: @override protected map<string, string> getparams() { jsonobject jsonobjectmembers=new jsonobject(); (int i=0; i<arr_added_userids.size(); i++) { seek { jsonobjectmembers.put("params_"+i,arr_added_userids.get(i)); } grab (jsonexception e) { e.printstacktrace(); } } map<string, string> params = new hashmap<string, string>(); params.put("host", session_userid); params.put("params",jsonobjectmembers.tostring()); homecoming params; } this tried: foreach($_post['params'] $key => $value) { seek { $stmt = $conn->prepare("insert crewmembers (crewid, member) values (:crewid, :member)"); $query_params = array( ':crewid' => $crewid, ':member' => $value ); ...
Read more

python - How to add an model instance to a django queryset? -

-
python - How to add an model instance to a django queryset? - it seems django queryset behaves somehow python list. but doesn't back upwards list's .append() method know. what want like: from my_django_app.models import mymodel queryset = mymodel.objects.none() queryset.append(mymodel.objects.first()) ## no list's .append() method! is there way add together model instance existing queryset? no. queryset representation of query - hence name - not arbitrary collection of instances. if need actual queryset rather list, seek accumulating ids of objects need , getting objects via __in query: list_of_ids = [] list_of_ids.append(my_id) ... queryset = mymodel.objects.filter(id__in=list_of_ids) this isn't efficient, though. python django
Read more

angularjs - No 'Access-Control-Allow-Origin' error from local domain to public API -

-
angularjs - No 'Access-Control-Allow-Origin' error from local domain to public API - i'm trying create app retrieve stock name, asking , purchase cost yahoo finance api, using restangular. i'm having problem accessing public api in local application i'm creating when request. url parameters i'm sending: http://finance.yahoo.com/d/quotes.csv?s=msft+ge&f=nab using postman, request returns array contains stock name, asking cost , purchase cost of stock. doing request in postman returns desired info without errors. when request in local, next error: xmlhttprequest cannot load http://finance.yahoo.com/d/quotes.csv?s=msft+ge&f=nab. no 'access-control-allow-origin' header nowadays on requested resource. origin 'http://localhost' hence not allowed access. the html code follows: <html ng-app="app"> <head> <title>restangular</title> <script src="https://ajax.googleapis.com/ajax/li...
Read more