awk의 구분자 사이의 개행

Question

nginx GET 요청 본문이 포함 된 파일을 구문 분석하고 있습니다. 그리고 때로는 동일한 요청의 두 부분 사이에 줄 바꿈이 포함되어 있기 때문에 이러한 요청을 awk으로 구문 분석 할 수 없습니다.

두 구분 기호가 awk -F'delimeter1: |delimiter2'이고 어쩌면 그 구분 기호 사이에 줄 바꿈이있을 수 있다고 awk에게 말할 수 있습니다. 그래서이 두 줄을 하나로 처리 할 수 ​​있습니까?

미리 감사드립니다.

샘플 입력 (자바의 오차가 임의의 일례이다)

[2017-12-04 20:53:07] [ERROR] [ID-XX] Get: sr=342x487&c64=(not set)&c1=Phones, MP3s, GPS&v=1&c33=427&d28= 
Like 
&je=0&s4d=4-b&c32=(not set)&ua=Opera/9.80 (Android; Opera Mini/32.0.2254/77.161; U; uk) Presto/2.12.423 Version/12.16&time=04/Dec/2017:20:52:02 +0200&qtype=get 
com.test.app. java.lang.StringIndexOutOfBoundsException: String index out of range: -1 
      at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75) 
      at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93) 
      at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359) 
      at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350) 
      at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302) 
      at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113) 
      at javax.swing.JComponent.getFontMetrics(JComponent.java:1626) 
      at javax.swing.text.WrappedPlainView.updateMetrics(WrappedPlainView.java:318) 
      at javax.swing.text.WrappedPlainView.updateChildren(WrappedPlainView.java:297) 
      at javax.swing.text.WrappedPlainView.insertUpdate(WrappedPlainView.java:463) 
      at javax.swing.plaf.basic.BasicTextUI$RootView.insertUpdate(BasicTextUI.java:1610) 
      at javax.swing.plaf.basic.BasicTextUI$UpdateHandler.insertUpdate(BasicTextUI.java:1869) 
      at javax.swing.text.AbstractDocument.fireInsertUpdate(AbstractDocument.java:201) 
      at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:748) 
      at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:707) 
      at javax.swing.text.PlainDocument.insertString(PlainDocument.java:130) 
      at javax.swing.text.DefaultEditorKit.read(DefaultEditorKit.java:273) 
      at javax.swing.JEditorPane.setText(JEditorPane.java:1416) 
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 
      at java.lang.String.substring(String.java:1967) ~[?:1.8.0_151] 
      ... 12 more 
[2017-12-04 21:03:07] [ERROR] [ID-YY] Get: sr=342x487&c64=(not set)&c1=Phones, MP3s, GPS&v=1&em=Exception: Error: [$sc:ind] Aborting!&ua=Opera/9.80 (Android; Opera Mini/30.0.2254/77.161; U; ru) Presto/2.12.423 Version/12.16&time=04/Dec/2017:21:03:07 +0200&qtype=get 
com.test.app. java.lang.StringIndexOutOfBoundsException: String index out of range: -1 
      at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75) 
      at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93) 
      at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359) 
      at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350) 
      at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302) 
      at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113) 
      at javax.swing.JComponent.getFontMetrics(JComponent.java:1626) 
      at javax.swing.text.WrappedPlainView.updateMetrics(WrappedPlainView.java:318) 
      at javax.swing.text.WrappedPlainView.updateChildren(WrappedPlainView.java:297) 
      at javax.swing.text.WrappedPlainView.insertUpdate(WrappedPlainView.java:463) 
      at javax.swing.plaf.basic.BasicTextUI$RootView.insertUpdate(BasicTextUI.java:1610) 
      at javax.swing.plaf.basic.BasicTextUI$UpdateHandler.insertUpdate(BasicTextUI.java:1869) 
      at javax.swing.text.AbstractDocument.fireInsertUpdate(AbstractDocument.java:201) 
      at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:748) 
      at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:707) 
      at javax.swing.text.PlainDocument.insertString(PlainDocument.java:130) 
      at javax.swing.text.DefaultEditorKit.read(DefaultEditorKit.java:273) 
      at javax.swing.JEditorPane.setText(JEditorPane.java:1416) 
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 
      at java.lang.String.substring(String.java:1967) ~[?:1.8.0_151] 
      ... 12 more 
[2017-12-04 19:40:02] [ERROR] [ID-ZZ] Get: el=search&dl=https://market.com/?dt=Market – Electronics Store | Web Store (Market.com)&id=104777577&a=770227875&t=pageview&ua=Mozilla/5.0 (Linux; Android 7.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36&time=04/Dec/2017:19:39:04 +0200&qtype=get 
com.test.app. java.lang.StringIndexOutOfBoundsException: String index out of range: -1 
      at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75) 
      at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93) 
      at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359) 
      at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350) 
      at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302) 
      at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113) 
      at javax.swing.JComponent.getFontMetrics(JComponent.java:1626) 
      at javax.swing.text.WrappedPlainView.updateMetrics(WrappedPlainView.java:318) 
      at javax.swing.text.WrappedPlainView.updateChildren(WrappedPlainView.java:297) 
      at javax.swing.text.WrappedPlainView.insertUpdate(WrappedPlainView.java:463) 
      at javax.swing.plaf.basic.BasicTextUI$RootView.insertUpdate(BasicTextUI.java:1610) 
      at javax.swing.plaf.basic.BasicTextUI$UpdateHandler.insertUpdate(BasicTextUI.java:1869) 
      at javax.swing.text.AbstractDocument.fireInsertUpdate(AbstractDocument.java:201) 
      at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:748) 
      at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:707) 
      at javax.swing.text.PlainDocument.insertString(PlainDocument.java:130) 
      at javax.swing.text.DefaultEditorKit.read(DefaultEditorKit.java:273) 
      at javax.swing.JEditorPane.setText(JEditorPane.java:1416) 
    Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1 
      at java.lang.String.substring(String.java:1967) ~[?:1.8.0_151] 
      ... 12 more 

원하는 출력 (인쇄 ID 및 바디() ","하나 개의 라인과 _&_으로 & 교체)

ID-XX "sr=342x487_&_c64=(not set)_&_c1=Phones, MP3s, GPS_&_v=1_&_c33=427_&_d28=Like_&_je=0_&_s4d=4-b_&_c32=(not set)_&_ua=Opera/9.80 (Android; Opera Mini/32.0.2254/77.161; U; uk) Presto/2.12.423 Version/12.16_&_time=04/Dec/2017:20:52:02 +0200_&_qtype=get" 
ID-YY "sr=342x487_&_c64=(not set)_&_c1=Phones, MP3s, GPS_&_v=1_&_em=Exception: Error: [$sc:ind] Aborting!_&_ua=Opera/9.80 (Android; Opera Mini/30.0.2254/77.161; U; ru) Presto/2.12.423 Version/12.16_&_time=04/Dec/2017:21:03:07 +0200_&_qtype=get" 
ID-ZZ "el=search_&_dl=https://example.market.com/?dt=Market – Electronics Store | Web Store (Market.com)_&_id=104777577_&_a=770227875_&_t=pageview_&_ua=Mozilla/5.0 (Linux; Android 7.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36_&_time=04/Dec/2017:19:39:04 +0200_&_qtype=get" 

이러한 찢어진 요청 본문 문자열은 많지 않습니다. 대부분 예상 한대로 한 줄에 있습니다. 또한 오류가있는 GET 요청 만 있으므로 검색 패턴에 Get을 포함 할 필요는 없습니다 (꼭 필요한 것은 아닙니다).

Answer 1

Awk 용액 :

awk 'f{ if (/^\[/) { printf "\042\n"; f=0 } else printf("%s", $0) } 
    /Get:/{ 
     f=1; gsub(/[\[\]]/, "", $4); id=$4; sub(/^.* Get: /, ""); 
     gsub("&", "_&_"); printf "%s \042%s",id,$0 
    } 
    END{ if (f) printf "\042\n" }' file 

• / Get:/-조우의 "Get 요청" 라인
  • f=1-f 이너/하위 처리
  • id=$4를 나타내는 마커 - 기음 apturing ID 필드 (예 :

ID-XX) 출력 :

ID-XX "sr=342x487_&_c64=(not set)_&_c1=Phones, MP3s, GPS_&_v=1_&_c33=427_&_d28=Like&je=0&s4d=4-b&c32=(not set)&ua=Opera/9.80 (Android; Opera Mini/32.0.2254/77.161; U; uk) Presto/2.12.423 Version/12.16&time=04/Dec/2017:20:52:02 +0200&qtype=get" 
ID-YY "sr=342x487_&_c64=(not set)_&_c1=Phones, MP3s, GPS_&_v=1_&_em=Exception: Error: [$sc:ind] Aborting!_&_ua=Opera/9.80 (Android; Opera Mini/30.0.2254/77.161; U; ru) Presto/2.12.423 Version/12.16_&_time=04/Dec/2017:21:03:07 +0200_&_qtype=get" 
ID-ZZ "el=search_&_dl=https://market.com/?dt=Market – Electronics Store | Web Store (Market.com)_&_id=104777577_&_a=770227875_&_t=pageview_&_ua=Mozilla/5.0 (Linux; Android 7.0; RNE-L21 Build/HUAWEIRNE-L21) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36_&_time=04/Dec/2017:19:39:04 +0200_&_qtype=get" 

Answer 2

난 당신이 오류가있는 라인을 유지하고 서식을 지정할 이해합니다.

구분 기호가 무엇인지 알 수 없습니다.

줄 끝에서 계속 지켜 보는 것이 이상합니다.

이이 John1024 @, 귀하의 관심

sed ' 
/.*ERROR] \[/!d      # get the line with ERROR 
s///        # delete all from start to ID 
:A 
/=get$/!{N;bA}      # if the line not end with =get; get one more 
s/\([^]]*\)[^:]*: \(.*\)/\1 "\2"/ # remove Get: and add " 
s/\n//g        # remove \n 
s/&/_&_/g       # replace & by _&_ 
' infile