hadoop - FIles comparing field by field in PIG -

-


hadoop - FIles comparing field by field in PIG -

i have 2 files like

file 1

id,sal,location,code 1000,1000,jupiter,f 1001,2000,jupiter,f 1002,3000,jupiter,f 1003,4000,jupiter,f 1004,5000,jupiter,f

file 2

id,sal,location,code 1000,2000,jupiter,f 1001,2000,jupiter,z 1002,3000,jupiter,f 1003,4000,jupiter,f 1004,5000,jupiter,f

when compare file1 file 2, need output like

1000, sal 1001,code

basically, should tell me field changed previous file along id. can done in pig.

you can solve problem challenging part output format mentioned. requires little bit complex logic output format.

i have fixed of border cases can check input create sure works combinations.

file1:

1000,1000,jupiter,f 1001,2000,jupiter,f 1002,3000,jupiter,f 1003,4000,jupiter,f 1004,5000,jupiter,f

file2:

1000,2000,jupiter,f 1001,2000,jupiter,z 1002,3000,jupiter,f 1003,4000,jupiter,f 1004,5000,jupiter,f

pigscript:

= load 'file1' using pigstorage(',') (id,sal,location,code); b = load 'file2' using pigstorage(',') (id,sal,location,code); c = bring together id,b id; d = foreach c generate a::id id,((a::sal == b::sal)?'':'sal') sal, ((a::location == b::location)?'':'location') location, ((a::code == b::code)?'':'code') code; --remove mutual fields between 2 files e = filter d not (sal=='' , location=='' , code==''); --the below 2 lines used formatting output f = foreach e generate id,replace(bagtostring(tobag(sal,location,code),','),'(,,$|,$)','') finaloutput; g = foreach f generate id,replace(finaloutput,',,',','); dump g;

output:

(1000,sal) (1001,code)

hadoop mapreduce apache-pig

Comments

Post a Comment

Popular posts from this blog

python - How to add an model instance to a django queryset? -

-
python - How to add an model instance to a django queryset? - it seems django queryset behaves somehow python list. but doesn't back upwards list's .append() method know. what want like: from my_django_app.models import mymodel queryset = mymodel.objects.none() queryset.append(mymodel.objects.first()) ## no list's .append() method! is there way add together model instance existing queryset? no. queryset representation of query - hence name - not arbitrary collection of instances. if need actual queryset rather list, seek accumulating ids of objects need , getting objects via __in query: list_of_ids = [] list_of_ids.append(my_id) ... queryset = mymodel.objects.filter(id__in=list_of_ids) this isn't efficient, though. python django
Read more

Using Android Volley to post an array to PHP -

-
Using Android Volley to post an array to PHP - this how post values volley: @override protected map<string, string> getparams() { jsonobject jsonobjectmembers=new jsonobject(); (int i=0; i<arr_added_userids.size(); i++) { seek { jsonobjectmembers.put("params_"+i,arr_added_userids.get(i)); } grab (jsonexception e) { e.printstacktrace(); } } map<string, string> params = new hashmap<string, string>(); params.put("host", session_userid); params.put("params",jsonobjectmembers.tostring()); homecoming params; } this tried: foreach($_post['params'] $key => $value) { seek { $stmt = $conn->prepare("insert crewmembers (crewid, member) values (:crewid, :member)"); $query_params = array( ':crewid' => $crewid, ':member' => $value ); ...
Read more

angularjs - No 'Access-Control-Allow-Origin' error from local domain to public API -

-
angularjs - No 'Access-Control-Allow-Origin' error from local domain to public API - i'm trying create app retrieve stock name, asking , purchase cost yahoo finance api, using restangular. i'm having problem accessing public api in local application i'm creating when request. url parameters i'm sending: http://finance.yahoo.com/d/quotes.csv?s=msft+ge&f=nab using postman, request returns array contains stock name, asking cost , purchase cost of stock. doing request in postman returns desired info without errors. when request in local, next error: xmlhttprequest cannot load http://finance.yahoo.com/d/quotes.csv?s=msft+ge&f=nab. no 'access-control-allow-origin' header nowadays on requested resource. origin 'http://localhost' hence not allowed access. the html code follows: <html ng-app="app"> <head> <title>restangular</title> <script src="https://ajax.googleapis.com/ajax/li...
Read more